Marker placement information estimating method and information processing device

ABSTRACT

A marker placement information estimating method includes a constraint condition setting step for setting a constraint condition that can be defined with a geometric condition regarding placement information of markers, a capturing step for capturing photographed images including a marker image, a marker detecting step for detecting markers from the photographed images, and an estimating step for estimating the placement information of the markers using the positions of the detected markers on the images and the constraint condition. Thus, even if a constraint condition regarding the placement of markers exists, the placement information can be obtained with precision so as to satisfy the constraint condition.

BACKGROUND OF THE INVENTION

1. Field of the Invention

The present invention relates to a method and device for estimating theplacement information of a marker disposed within a 3-D space.

2. Description of the Related Art

In recent years, studies regarding MR (Mixed Reality) technology havebeen actively pursued. The MR technology is used for seamlesslyintegrating real space with virtual space created by a computer. Of theMR technology, AR (Augmented Reality; also referred to as enhancedreality) technology which superimposes virtual space over real space hasparticularly attracted attention.

An image display device on which the AR technology displays an image isrealized by a video see-through method which displays a synthesizedimage obtained by superimposing a computer-generated image concernedwith virtual space (a virtual object, character information, or thelike, drawn with computer graphics) rendered according to the positionand orientation of a later-described imaging device over an image ofreal space photographed using an imaging device such as a video camera,or by an optical see-through method which displays an image of virtualspace rendered according to an observer's viewing position andorientation on an optical-see-through-type display mounted on theobserver's head.

There are expectations for application of the AR technology to variousfields such as surgical aids which superimposes the state within apatient's body over the surface of the body, an architectural simulationwhich superimposes a virtual building over an image of vacant land,assembly aids which superimposes assembly procedures and wiring forassembly of machines or other devices, and the like.

The most important problem to be solved in the AR technology is how toaccurately perform the registration between real space and virtualspace, and heretofore, many methods have been attempted. Theregistration problem in the AR technology corresponds to a problem ofobtaining the position and orientation of an imaging device in a scene(i.e., in the reference coordinate system) in the case of the videosee-through method. Similarly, in the case of the optical see-throughmethod, the registration problem corresponds to a problem for obtainingthe position and orientation of an observer or a display device in thescene.

A commonly-employed method for solving the former problem is to disposeartificial markers or make natural characteristic points markers in thescene, based on the correspondence between the projected positions ofthe markers within an image photographed by an imaging device and thepositions in the reference coordinate system of the markers, so as toobtain the position and orientation of the imaging device in thereference coordinate system. Also, a commonly-employed method forsolving the latter problem is to mount the imaging device on a target tobe measured (e.g., an observer's head or display), with the position andorientation of the imaging device being obtained in the same way as withthe former method, and the position and orientation of the target to bemeasured is obtained based thereupon.

Description will be made regarding a conventional example of aposition-and-orientation measuring device for measuring the position andorientation of a later-described imaging device by correlating the 2-Dcoordinates of a marker to be detected from an image photographed by animaging device and the 3-D position of the marker in the referencecoordinate system with reference to FIG. 1. As illustrated in FIG. 1, aposition-orientation measuring device 100 in the present conventionalexample includes a marker detecting unit 110 and position-orientationcalculating unit 120, and is connected to an imaging device 130.

Also, K markers Q_(k) (k=1, 2, and so on through K) of which thepositions in the reference coordinate system are known are disposed inreal space as the markers for obtaining the position and orientation ofthe imaging device 130. The example in FIG. 1 illustrates a situation inwhich four markers Q₁, Q₂, Q₃, and Q₄ are disposed. Of these, threemarkers Q₁, Q₃, and Q₄ are inside the field of view of the imagingdevice 130 and one marker Q₂ is outside the field of view of the imagingdevice 130.

The markers Q_(k) can be any shape, such as a circular marker having adifferent color from other markers, or the like, as long as theprojected position of a marker within a photographed image can bedetected, and also the marker can be identified. For example, naturalcharacteristic points within 3-D space may be employed, and such pointsmay be detected within a photographed image using template matching. Animage output from the imaging device 130 is input to theposition-orientation measuring device 100. The marker detecting unit 110inputs an image by the imaging device 130, and detects the imagecoordinates of the markers Q_(k) photographed on the image. For example,in the event that each of the markers Q_(k) is made up of a circularmarker each having a different color, the marker detecting unit 110detects a region corresponding to each marker color from on the inputimage, and takes the barycentric position as the detected coordinates ofthe marker.

Further, the marker detecting unit 110 outputs the image coordinatesu^(Mkn) of each detected marker Q_(kn) and the identifier k_(n) thereofto the position-orientation calculating unit 120. Here, n (=1, 2, and soon through N) is a symbol representing the serial number of the detectedmarkers, and N represents the total number of the detected markers. Forexample, in the case of FIG. 1, N=3, so the identifiers k₁=1, k₂=3, k₃=4and the image coordinates u^(Mk1), u^(Mk2), and u^(Mk3) corresponding tothese are output.

The position-orientation calculating unit 120 calculates the positionand orientation of the imaging device 130 based on the correlationbetween the image coordinates u^(Mkn) of each detected marker Q_(kn) andthe position in the reference coordinate system of the marker Q_(kn),which is held as known information beforehand. A method for calculatingthe position and orientation of an imaging device based on a pair of the3-D coordinates of a marker and image coordinates has been proposed inthe field of photogrammetry as of old (for example, see R. M. Haralick,C. Lee, K. Ottenberg, and M. Nolle: “Review and analysis of solutions ofthe three point perspective pose estimation problem”, Int'l. J. ComputerVision, vol. 13, no. 3, pp. 331-356, 1994 and M. A. Fischler and R. C.Bolles: “Random sample consensus: a paradigm for model fitting withapplications to image analysis and automated cartography”, Comm. ACM,vol. 24, no. 6, pp. 381-395, 1981). The position-orientation calculatingunit 120 calculates the position and orientation of the imaging device130 using the method described in R. M. Haralick, C. Lee, K. Ottenberg,and M. Nolle: “Review and analysis of solutions of the three pointperspective pose estimation problem”, Int'l. J. Computer Vision, vol.13, no. 3, pp. 331-356, 1994, for example.

Note that description has been made regarding the case of employingmarkers (hereinafter, referred to as “point markers”) as multiple pointswithin 3-D space, but a calculation method for calculating the positionand orientation of an imaging device using square-shaped markers(hereinafter, referred to as “square markers”) having a known size hasbeen proposed as disclosed in J. Rekimoto: “Configuration method ofaugmented reality using 2-D matrix code”, Interactive System andSoftware IV, Kindai Kagakusha, 1996 and Kato, M. Billinghurst, Asano andTachibana: “Augmented reality based on marker tracing and calibrationthereof”, Japan Virtual Reality Academic Journal, vol. 4, no. 4, pp.607-616, 1999, for example. A calculation method of the position andorientation of an imaging device using combination of square markers andpoint markers has been proposed, as disclosed in H. Kato, M.Billinghurst, I. Poupyrev, K. Imamoto and K. Tachibana: “Virtual objectmanipulation on a table-top AR environment”, Proc. ISAR2000, pp.111-119, 2000, for example. With this calculation method, point markershave an advantage wherein point markers can set even in a narrow place,square markers have advantages wherein identification is easy, and theposition and orientation of the imaging device can be obtained from onlyone marker since one marker includes sufficient information, thusutilizing these two types of markers in a complementary manner.

According to the aforementioned methods, based on an image photographedby an imaging device, the position and orientation of the imaging devicehas been acquired since the past.

On the other hand, an arrangement has been made in which a6-degree-of-freedom position and orientation sensor such as a magneticsensor, ultrasonic sensor, or the like is attached to an imaging deviceserving as a target to be measured, and the position and orientation theimaging device is measured by concomitant use with marker detection byimage processing as described above, as disclosed in Japanese PatentLaid-Open No. 11-084307, Japanese Patent Laid-Open No. 2000-041173, andA. State, G. Hirota, D. T. Chen, W. F. Garrett and M. A. Livingston:“Superior augmented reality registration by integrating landmarktracking and magnetic tracking”, Proc. SIGGRAPH'96, pp. 429-438, 1996.The accuracy of a sensor output changes depending on a measuring range,but can be obtained robustly, so a method using both sensor and imageprocessing can improve robustness as compared with a method using imageprocessing alone.

With a registration method using markers, the position in the referencecoordinate system in the case of point markers and the position andorientation in the reference coordinate system in the case of squaremarkers needs to be known for obtaining the position and orientation inthe reference coordinate system of an imaging device serving as a targetto be measured. In the case of a square marker, the square marker itselfis often taken as the reference of the coordinate system withoutseparately providing the reference coordinate system, but in the case ofemploying multiple square markers, the mutual position and orientationrelations need to be known, and accordingly, there is no difference inthat the reference coordinate system needs to be employed.

The position and orientation of a marker may be measured by hand using ameasuring tape, ruler, or protractor, or by a surveying instrument, butmeasurement techniques utilizing images have been performed to improveaccuracy and save time. The position of a point marker can be measuredby a method called the bundle adjustment method. The bundle adjustmentmethod is a method in which a great number of point markers arephotographed by an imaging device, the position and orientation of theimaging device taking each image and the positions of point markers areobtained by repeated calculation so that the error between the projectedpositions where the markers are actually observed on the image, and theprojected positions to be calculated from the position and orientationof the imaging device, and the positions of the markers, can beminimized.

Also, a method for measuring the position and orientation of multiplesquare markers disposed within 3-D space has been disclosed in G.Baratoff, A. Neubeck and H. Regenbrecht: “Interactive multi-markercalibration for augmented reality applications”, Proc. ISMAR2002, pp.107-116, 2002. With G. Baratoff, A. Neubeck and H. Regenbrecht:“Interactive multi-marker calibration for augmented realityapplications”, Proc. ISMAR2002, pp. 107-116, 2002, the position andorientation of an imaging device taking each image, and the position andorientation of each square marker are obtained using a method in which agreat number of images of multiple square markers disposed within 3-Dspace are photographed, and repeated calculation is performed so thatprojection error can be minimized.

With the aforementioned conventional method for measuring the positionand orientation of a marker, in the case that there is a constraintcondition regarding the positions of markers such that multiple markersare on the same plane, the measured results cannot satisfy theconstraint condition in some cases. This is because the aforementionedmeasuring method is a method for minimizing the projection error ofmarkers on the photographed image, and accordingly, in the event thatthe parameters (the focal length of a camera, the position of aprincipal point, and a lens distortion correction parameter) employedfor calculation of the projected position of a marker, and theobservation position of a marker on an image include a margin of error,the measured values also include the margin of error. Consequently,measurement results are obtained that multiple markers, which should beon the same plane, are not on the same plane.

SUMMARY OF THE INVENTION

The present invention has been made in light of the aforementionedproblems, and provides a method for obtaining the position andorientation of a marker so as to satisfy the constraint condition in theevent there is a constraint condition regarding the placement ofmarkers.

To this end, according to a first aspect of the present invention, amarker placement information estimating method for estimating spatialplacement information of markers present within space under a geometricconstraint includes: a marker detecting step for detecting markers froma photographed image including markers; a constraint condition settingstep for setting a constraint condition of markers; a marker projectedposition calculating step for calculating a projected position where amarker is projected on an image surface based on the approximate valueof the position and orientation of an imaging unit of the photographedimage and the approximate value of the placement information of themarker; a correction value calculating step for obtaining a correctionvalue for correcting the placement information of a marker so as toreduce a sum of error between the position of the marker on the imagesurface detected in the marker detecting step and the projected positionof the marker on the image surface obtained in the marker projectedposition calculating step, and also so as to satisfy the constraintcondition of markers set in the constraint condition setting step; aplacement information correcting step for correcting the placementinformation of a marker based on the correction value obtained in thecorrection value calculating step; and a repeated calculating step forcalculating the placement information of the marker by repeatedlyperforming the marker projected position calculating step, thecorrection value calculating step, and the placement informationcorrecting step using the placement information of the marker correctedin the placement information correcting step instead of the projectedposition calculated in the marker projected position calculating step.

According to a second aspect of the present invention, a markerplacement information estimating method for estimating the spatialplacement information of markers from a photographed image includes: aconstraint condition setting step for setting a constraint conditionthat can be defined with a geometric condition relating to the placementinformation of markers; a capturing step for capturing photographedimages including marker images; a marker detecting step for detectingmarkers from the photographed images; and an estimating step forestimating the placement information of the markers based on thepositions of the markers detected on the images and the constraintcondition.

Other features and advantages of the present invention will be apparentfrom the following description of exemplary embodiments taken inconjunction with the accompanying drawings, in which like referencecharacters designate the same or similar parts throughout the figuresthereof.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 is a diagram illustrating a conventional position-and-orientationmeasuring device.

FIG. 2 is a block diagram illustrating the functional configuration of amarker position-and-orientation estimating device according to a firstembodiment.

FIG. 3 is a block diagram illustrating the hardware configuration of themarker position-and-orientation estimating device according to the firstembodiment.

FIG. 4A is a diagram illustrating a point marker employed in the firstembodiment.

FIG. 4B is a diagram illustrating a plane marker employed in the firstembodiment.

FIG. 5 is a diagram illustrating a GUI (Graphical User Interface) forsetting multiple markers present on the same plane.

FIG. 6 is a diagram illustrating a camera coordinate system and an imagecoordinate system.

FIG. 7 is a diagram illustrating a collinear condition equation.

FIG. 8 is a diagram illustrating a plane definition method.

FIG. 9 is a flowchart illustrating procedures for obtaining thepositions of point markers, and the positions and orientations of squaremarkers, according to the first embodiment.

FIG. 10 is a diagram illustrating a case in which two planesconstraining markers exist.

FIG. 11 is a diagram illustrating marker placements to which a thirdembodiment can be applied.

FIG. 12 is a diagram illustrating marker placements to which a fourthembodiment can be applied.

FIG. 13 is a diagram illustrating a marker being constrained on the sideface of a cylinder.

FIG. 14 is a diagram illustrating a marker being constrained on aspherical surface.

FIGS. 15A through 15C are diagrams illustrating a GUI according to thefourth embodiment.

DESCRIPTION OF THE EMBODIMENTS

Description is made below regarding exemplary embodiments of the presentinvention with reference to the appended drawings.

First Embodiment

FIG. 2 is a block diagram illustrating the functional configuration of amarker position-and-orientation estimating device 2 according to thepresent embodiment. An imaging unit 210 is a camera, which photographs ascene in which markers are disposed. An image capturing unit 220 inputsan image photographed by the imaging unit 210 to a computer. A markerdetecting-and-identifying unit 230 detects markers from the image inputto the computer from the image capturing unit 220, and identifies eachdetected marker. A marker placement condition setting unit 240 sets aconstraint condition regarding placement of markers. A markerposition-and-orientation calculating unit 250 calculates the positionand orientation of each marker so as to satisfy the constraint conditionregarding the position of markers set by the marker placement conditionsetting unit 240 based on the marker detection results of the markerdetecting-and-identifying unit 230.

FIG. 3 is a block diagram illustrating the hardware configuration of themarker position-and-orientation estimating device 2 according to thepresent embodiment. The hardware configuration illustrated in FIG. 3 isthe same configuration as that of a common personal computer, but isfurther connected with an image capturing device 310 and a camera 311.The image capturing device 310 is for inputting an image photographed bythe camera 311 to the computer, and corresponds to the image capturingunit 220. An example of the image capturing device 310 is a videocapturing board, but is not restricted to this as long as an imagephotographed by a camera can be input. A CPU 301 serves as the markerdetecting-and-identifying unit 230, marker placement condition settingunit 240, and marker position-orientation calculating unit 250 byexecuting a program stored in an unshown optical medium, externalstorage device, or the like.

The hardware-configuration of the marker position-and-orientationestimating device 2 shown in FIG. 3 may also include other conventionalcomputer components, such as a graphic accelerator 302 in communicationwith a head-mounted display (HMD) 304, a storage device, such as a disk305, a read-only memory (ROM) 306, a random access memory (RAM) 307, andone or more input devices, such as a keyboard 309 and a mouse 308.

Next, description is made regarding the overview of operation of themarker position-and-orientation estimating device according to thepresent embodiment having the aforementioned configuration.

First, description is made regarding markers to be employed with thepresent embodiment. FIGS. 4A and 4B are diagrams illustrating markers tobe employed with the present embodiment. FIG. 4A illustrates a pointmarker in which the placement information is represented by the positionof a point in 3-D space. Let us say that point markers have a circularshape having a single color so as to be detected within an image, andthe center of gravity of each point marker represents the positionthereof in 3-D space. FIG. 4B illustrates a plane marker of which theplacement information is represented by the position and orientation ofa plane in 3-D space. The present embodiment employs square markers asplane markers, but plane markers are not restricted to square markers,so any shaped markers may be employed as long as markers have a form ofa planar shape. Let us say that the center of a square, i.e., theintersecting point of two diagonal lines, represents the position of asquare marker in 3-D space. As illustrated in FIG. 4B, let us say thatthe normal line direction of the square marker is taken as the z axis,the coordinate system of the square marker is assigned so that the xaxis and y axis can be parallel to the two sides, and the orientation ofthe coordinate system of the square marker to the reference coordinatesystem is taken as the orientation of the square marker. The squaremarkers according to the present embodiment are surrounded with a blackframe so as to be easily detected on an image, so a square within theblack frame is employed as a square marker.

(1. Photographing of Markers)

A user photographs many images of a scene in which markers are disposedusing the imaging unit 210. The photographed images are input to thecomputer by the image capturing unit 220.

(2. Detection and Identification of Markers)

The marker detecting-and-identifying unit 230 detects markers from thephotographed images input to the computer, and performs identificationthereof. Here, the term “detection of a marker” means to obtain theimage coordinates of a marker on a 2-D image. In the case of pointmarkers, the image coordinates of the center of gravity of marker regionon an image are obtained, and, in the case of square markers, the imagecoordinates of each vertex are obtained. The image coordinates of amarker may be obtained automatically, or may be manually specified, forexample, by the user clicking on the image with a mouse. In the case ofobtaining the position of a point marker automatically, for example, apixel having the same color as a marker is extracted based on whether ornot the pixel belongs to the specific region of a YUV color space imagecorresponding to the color, a marker region is obtained by labeling thepixels connected as the same color, and the center of gravity of themarker region within the image is calculated. Also, in the case ofobtaining the position of each vertex of a square marker automatically,for example, an input image is converted into binarized image, the blackframe region of a square marker is detected, and a quadrangle inscribingthe black frame region is obtained.

The term “identification of a marker” means to identify a marker to bedetected within an image as a unique marker. Identification of a markermay be performed automatically, or may be performed manually by theuser. In the case of identifying a point marker automatically, forexample, multiple markers having different colors are employed, andidentification is performed based on the labeling color of the markerregion at the time of detection of markers. Also, in the case ofidentifying a square marker automatically, for example, identificationis performed by giving a unique pattern to each marker.

(3. Setting of a Marker Placement Condition)

The user sets a constraint condition regarding the placement of markersusing the marker placement condition setting unit 240. With the presentembodiment, a constraint condition that multiple markers exist on thesame plane is given as a constraint condition regarding the placement ofmarkers. Let us say that a plane employed here is an unknown plane, andmultiple unknown planes exist. FIG. 5 illustrates a GUI (Graphical UserInterface) for setting multiple markers present on the same plane. Aphotographed image switching portion exists on the GUI, where an imagephotographed by the imaging unit is displayed. Display of a photographedimage may be switched with a button on the photographed image switchingportion. Upon the user clicking a Previous button with the mouse, theprevious photographed image is displayed. Upon the user clicking a Nextbutton, the subsequent photographed image is displayed. The user setsmarkers on the same plane while watching the photographed image. On thephotographed image, the name unique to each marker is displayed neareach identified marker. FIG. 5 illustrates the case that a photographedimage in which two point markers (POINT1 and POINT2 in the drawing) andtwo square markers (SQUARE1 and SQUARE2 in the drawing) are photographedis displayed on the photographed image display portion. Let us say thatof these markers, the point marker POINT1 and the square marker SQUARE1exist on the same plane. The user selects either POINT1 or SQUARE1within the photographed image by clicking near either marker with themouse. The name of the selected marker is changed to bold face. Next,the user forms a group from markers on the same plane by clicking nearbymarkers present on the same plane with the left mouse button as theselected marker while depressing the shift key. In the case of clickingon a marker belonging to a group present on the same plane, the namedisplay of all of the markers belonging to the group is changed to boldface. In this state, the marker can be added to the group by the userclicking another marker with the left mouse button while depressing theshift key. Also, the user clicking nearby a marker belonging to a groupwith the right mouse button while depressing the shift key releases themarker from the group, and the user clicking the same marker with theright mouse button without depressing the shift key releases the entiregroup.

(4. Calculation of the Position and Orientation of a Marker)

The position and orientation of a marker in the reference coordinatesystem is calculated based on the detection and identification resultsof the marker and the constraint condition regarding the placement ofmarkers. First, description is made regarding perspective projectiontransform. FIG. 6 is a diagram illustrating the camera coordinate systemand the image coordinate system. Let us say that the intersecting pointbetween the viewing axis and an image surface is taken as the originO_(i) of the image coordinate system, the horizontal direction of animage is taken as the x_(i) axis, and the vertical direction of theimage is taken as the y_(i) axis. Also, let us say that the focal lengthbetween the origin O_(c) of the camera coordinate system and the imagesurface is f, the z_(c) axis of the camera coordinate system is taken inthe opposite direction of the viewing axis, the x_(c) axis is taken inthe direction in parallel with the horizontal direction of the image,and the y_(c) axis is taken in the direction in parallel with thevertical direction of the image.

According to perspective projection transform, the point x_(c) =[x_(c)y_(c) z_(c)]^(t) on the camera coordinate system is projected on a pointof which the image coordinates are u=[u_(x) u_(y)]^(t) as shown inExpression (2). $\begin{matrix}{{u_{x} = {{- f}\frac{x_{c}}{z_{c}}}}{u_{y} = {{- f}\frac{y_{c}}{z_{c}}}}} & (2)\end{matrix}$

With the present embodiment, let us say that lens distortion does notexist, or has been corrected, and the camera is a pin-hole camera. Asillustrated in FIG. 7, Expression 1 represents that a point in space,the projected point of the point on an image, and a camera position(viewpoint) exist on the same straight line, and is also referred to asa collinear condition.

In the reference coordinate system, let us say that the camera positionis t=[t_(x) t_(y) t_(z)]^(t), and camera orientation (the orientation ofthe reference coordinate system to the camera coordinate system, inreality) is ω=[ω_(x) ω_(y) ω_(z)]. Note that ω is a 3-degree-of-freedomorientation expression, and orientation is represented with a rotationaxial vector and rotation angle. If the rotation angle is r_(a), thenr_(a) is represented with ω as shown in Expression (3).r _(a)={square root}{square root over (ω_(x) ²+ω_(y) ²+ω_(z) ²)}  (3)

Also, if the rotation axial vector is r_(axis)=[r_(x) r_(y) r_(z)]^(t),the relationship between r_(axis) and ω is represented as shown inExpression (4).[ω_(x) ω_(y) ω_(z) ]=[r _(a) r _(x) r _(a) r _(y) r _(a) r _(z)]  ( 4 )

The relationship between ω (rotation angle r_(a) and rotation axialvector r_(axis)) and 3×3 rotational transform matrix R is represented asshown in Expression (5). $\begin{matrix}{R = {\begin{bmatrix}R_{11} & R_{12} & R_{13} \\R_{21} & R_{22} & R_{23} \\R_{31} & R_{32} & R_{33}\end{bmatrix}\quad = \begin{bmatrix}{{r_{x}^{2}\left( {1 - {\cos\quad r_{a}}} \right)} + {\cos\quad r_{a}}} & {{r_{x}{r_{y}\left( {1 - {\cos\quad r_{a}}} \right)}} - {r_{z}\sin\quad r_{a}}} & {{r_{z}{r_{x}\left( {1 - {\cos\quad r_{a}}} \right)}} + {r_{y}\sin\quad r_{a}}} \\{{r_{x}{r_{y}\left( {1 - {\cos\quad r_{a}}} \right)}} + {r_{z}\sin\quad r_{a}}} & {{r_{y}^{2}\left( {1 - {\cos\quad r_{a}}} \right)} + {\cos\quad r_{a}}} & {{r_{y}{r_{z}\left( {1 - {\cos\quad r_{a}}} \right)}} - {r_{x}\sin\quad r_{a}}} \\{{r_{z}{r_{x}\left( {1 - {\cos\quad r_{a}}} \right)}} - {r_{y}\sin\quad r_{a}}} & {{r_{y}{r_{z}\left( {1 - {\cos\quad r_{a}}} \right)}} + {r_{x}\sin\quad r_{a}}} & {{r_{z}^{2}\left( {1 - {\cos\quad r_{a}}} \right)} + {\cos\quad r_{a}}}\end{bmatrix}}} & (5)\end{matrix}$

The camera coordinate x_(c) of a point x_(w)=[x_(w) y_(w) z_(w)]^(t) onthe reference coordinate system is represented with t and R as shown inExpression (6). $\begin{matrix}{\begin{bmatrix}x_{c} \\y_{c} \\z_{c}\end{bmatrix} = {{R\begin{bmatrix}{x_{w} - t_{x}} \\{y_{w} - t_{y}} \\{z_{w} - t_{z}}\end{bmatrix}} = {\begin{bmatrix}R_{11} & R_{12} & R_{13} \\R_{21} & R_{22} & R_{23} \\R_{31} & R_{32} & R_{33}\end{bmatrix}\begin{bmatrix}{x_{w} - t_{x}} \\{y_{w} - t_{y}} \\{z_{w} - t_{z}}\end{bmatrix}}}} & (6)\end{matrix}$

According to Expressions (2) and (6), i.e., perspective projectiontransform, the point x_(w)=[x_(w) y_(w) z_(w)]^(t) on the referencecoordinate system is projected on the point u=[u_(x) u_(y)]^(t) on theimage as shown in Expression (7). $\begin{matrix}{{u_{x} = {{{- f}\frac{x_{c}}{z_{c}}} = {{- f}\quad\frac{{R_{11}\left( {x_{w} - t_{x}} \right)} + {R_{12}\left( {y_{w} - t_{y}} \right)} + {R_{13}\left( {z_{w} - t_{z}} \right)}}{{R_{31}\left( {x_{w} - t_{x}} \right)} + {R_{32}\left( {y_{w} - t_{y}} \right)} + {R_{33}\left( {z_{w} - t_{z}} \right)}}}}}{u_{y} = {{{- f}\frac{y_{c}}{z_{c}}} - {f\quad\frac{{R_{21}\left( {x_{w} - t_{x}} \right)} + {R_{22}\left( {y_{w} - t_{y}} \right)} + {R_{23}\left( {z_{w} - t_{z}} \right)}}{{R_{31}\left( {x_{w} - t_{x}} \right)} + {R_{32}\left( {y_{w} - t_{y}} \right)} + {R_{33}\left( {z_{w} - t_{z}} \right)}}}}}} & (7)\end{matrix}$

Ideally, the projected position (calculated position) calculated fromExpression (7) based on t, ω, x_(w) is identical to the position(observation position) to be observed. Accordingly, if we say that theerror between the projected position and the observation position in thehorizontal direction of the image is F, the error in the verticaldirection is G, and the observation position is u_(o)=[u_(ox)u_(oy)]^(t), F and G become zero as shown in Expression (8).$\begin{matrix}{{F = {{{{- f}\quad\frac{{R_{11}\left( {x_{w} - t_{x}} \right)} + {R_{12}\left( {y_{w} - t_{y}} \right)} + {R_{13}\left( {z_{w} - t_{z}} \right)}}{{R_{31}\left( {x_{w} - t_{x}} \right)} + {R_{32}\left( {y_{w} - t_{y}} \right)} + {R_{33}\left( {z_{w} - t_{z}} \right)}}} - u_{ox}} = 0}}{G = {{{{- f}\quad\frac{{R_{21}\left( {x_{w} - t_{x}} \right)} + {R_{22}\left( {y_{w} - t_{y}} \right)} + {R_{23}\left( {z_{w} - t_{z}} \right)}}{{R_{31}\left( {x_{w} - t_{x}} \right)} + {R_{32}\left( {y_{w} - t_{y}} \right)} + {R_{33}\left( {z_{w} - t_{z}} \right)}}} - u_{oy}} = 0}}} & (8)\end{matrix}$

F and G are functions regarding the camera position t, cameraorientation ω, and position x_(w) of the observation target point on thereference coordinates. Here, the position x_(w) of the point on thereference coordinates becomes a different variable function depending onthe types of markers, and whether to constrain markers on the plane, soF and G also become a different variable function depending on the typesof markers, and whether to constrain markers on the plane. Descriptionis made below by dividing into two cases, i.e., the case of constrainingmarkers on the plane, and the case of not constraining markers on theplane.

(Case of Not Constraining Markers on the Plane)

In the case of point markers, if we say that the position of a pointmarker on the reference coordinate system is t_(pn)=[t_(pnx) t_(pny)t_(pnz)]^(t), x_(w) is identical with t_(pn) as shown in Expression (9).$\begin{matrix}{\begin{bmatrix}x_{w} \\y_{w} \\z_{w}\end{bmatrix} = \begin{bmatrix}t_{pnx} \\t_{pny} \\t_{pnz}\end{bmatrix}} & (9)\end{matrix}$

Accordingly, as shown in Expression (10), F and G become functionsregarding the camera position t, camera orientation ω, and positiont_(pn) of a point marker on the reference coordinate system.F(t _(x) ,t _(y) ,t _(z),ω_(x),ω_(y),ω_(z) ,t _(pnx) ,t _(pny) ,t_(pnz))=0G(t _(x) ,t _(y) ,t _(z),ω_(x),ω_(y),ω_(z) ,t _(pnx) ,t _(pny) ,t_(pnz))=0   (10)

Next, description is made regarding the case of square markers. Squaremarkers are represented with the position t_(s)=[t_(sx) t_(sy)t_(sz)]^(t) in the reference coordinate system and the orientationω_(s)=[ω_(sx) ω_(sy) ω_(sz)] as to the reference coordinate system (letus say that 3×3 rotational transform matrix corresponding to ω_(s) isR_(s)). Let us say that the position of vertices of a square marker inthe square marker coordinate system is x_(s)=[x_(s) y_(s) O]^(t). Theposition x_(w) of the vertices of a square marker in the referencecoordinate system becomes a function regarding t_(s) and ω_(s) (R_(s))as shown in Expression (11). $\begin{matrix}{\begin{bmatrix}x_{w} \\y_{w} \\z_{w} \\1\end{bmatrix} = {{\begin{bmatrix}R_{s} & t_{s} \\0 & 1\end{bmatrix}\begin{bmatrix}x_{s} \\y_{s} \\0 \\1\end{bmatrix}} = {\begin{bmatrix}R_{s11} & R_{s12} & R_{s13} & t_{sx} \\R_{s21} & R_{s22} & R_{s23} & t_{sy} \\R_{s31} & R_{s32} & R_{s33} & t_{sz} \\0 & 0 & 0 & 1\end{bmatrix}\begin{bmatrix}x_{s} \\y_{s} \\0 \\1\end{bmatrix}}}} & (11)\end{matrix}$

Accordingly, as shown in Expression (12), F and G become functionsregarding the camera position t, camera orientation ω, position t_(p) ofa square marker, and orientation ω_(s) of a square marker.F(t _(x) ,t _(y) ,t _(z),ω_(x),ω_(y),ω_(z) ,t _(sx) ,t _(sy) ,t_(sz),ω_(sx),ω_(sy),ω_(sz))=0G(t _(x) ,t _(y) ,t _(z),ω_(x),ω_(y),ω_(z) ,t _(sx) ,t _(sy) ,t_(sz),ω_(sx),ω_(sy),ω_(sz))=0   (12)(Case of Constraining Markers on the Plane)

First, description is made regarding a method for defining a plane. Asillustrated in FIG. 8, the plane is represented with polar coordinateexpression r=[r θ φ]. The z_(p) axis of the plane coordinate system isparallel to the normal line direction of the plane, and passes throughthe point of origin in the reference coordinate system. The orientationof the plane is defined with rotation θ around the y axis and rotation φaround the x axis. Also, r is a signed distance between the plane andthe origin o_(w) of the reference coordinate system. The rotation matrixR_(p) representing the orientation of the plane is represented as shownin Expression (13). $\begin{matrix}{R_{p} = {\begin{bmatrix}R_{p11} & R_{p12} & R_{p13} \\R_{p21} & R_{p22} & R_{p23} \\R_{p31} & R_{p32} & R_{p33}\end{bmatrix}\quad = {{\begin{bmatrix}1 & 0 & 0 \\0 & {\cos\quad\phi} & {{- \sin}\quad\phi} \\0 & {\sin\quad\phi} & {\cos\quad\phi}\end{bmatrix}\begin{bmatrix}{\cos\quad\theta} & 0 & {\sin\quad\theta} \\0 & 1 & 0 \\{{- \sin}\quad\theta} & 0 & {\cos\quad\theta}\end{bmatrix}}\quad = \begin{bmatrix}{\cos\quad\theta} & 0 & {\sin\quad\theta} \\{\sin\quad\theta\quad\sin\quad\phi} & {\cos\quad\phi} & {{- \cos}\quad\theta\quad\sin\quad\phi} \\{{- \sin}\quad{\theta cos}\quad\phi} & {\sin\quad\phi} & {\cos\quad{\theta cos}\quad\phi}\end{bmatrix}}}} & (13)\end{matrix}$Also, the position t_(pl)=[t_(plx) t_(ply) t_(plz)]^(t) of the point oforigin o_(p) of the plane coordinate system in the reference coordinatesystem is obtained as an intersecting point between the z_(p) axis andthe plane as shown in Expression (14).Note that {right arrow over (n)} represents the normal line vector ofthe plane. $\begin{matrix}{t_{pl} = {\begin{bmatrix}t_{plx} \\t_{ply} \\t_{plz}\end{bmatrix} = {{r \cdot \overset{->}{n}} = {{r \cdot {\begin{bmatrix}R_{p11} & R_{p12} & R_{p13} \\R_{p21} & R_{p22} & R_{p23} \\R_{p31} & R_{p32} & R_{p33}\end{bmatrix}\begin{bmatrix}0 \\0 \\1\end{bmatrix}}}\quad = \begin{bmatrix}{r\quad\sin\quad\theta} \\{{- r}\quad\cos\quad\theta\quad\sin\quad\phi} \\{r\quad\cos\quad{\theta cos}\quad\phi}\end{bmatrix}}}}} & (14)\end{matrix}$The position x_(w) of the point x_(p)=[x_(p) y_(p) O]^(t) on the planecoordinate system in the reference coordinate system represented withthe aforementioned polar coordinate expression r=[r θ φ] is representedas shown in Expression (15). As can be understood from Expression (15),x_(w) and x_(p) are functions of r. $\begin{matrix}{x_{w} = {\begin{bmatrix}x_{w} \\y_{w} \\z_{w}\end{bmatrix} = {{\begin{bmatrix}R_{p11} & R_{p12} & t_{plx} \\R_{p21} & R_{p22} & t_{ply} \\R_{p31} & R_{p32} & t_{plz}\end{bmatrix}\begin{bmatrix}x_{p} \\y_{p} \\1\end{bmatrix}}\quad = \begin{bmatrix}{{x_{p}\quad\cos\quad\theta}\quad + {r\quad\sin\quad\theta}} \\{{x_{p}\quad\sin\quad\theta\quad\sin\quad\phi} + {y_{p}\quad\cos\quad\phi}\quad - {r\quad\cos\quad\theta\quad\sin\quad\phi}} \\{{{- x_{p}}\quad\sin\quad\theta\quad\cos\quad\phi} + {y_{p}\quad\sin\quad\phi} + {r\quad\cos\quad\theta\quad\cos\quad\phi}}\end{bmatrix}}}} & (15)\end{matrix}$

The point marker constrained on the plane is represented with theposition t^(pl) _(p)=t^(pl) _(px) t^(pl) _(py) O]^(t). As shown inExpression (16), x_(p) is identical with t^(pl) _(p). $\begin{matrix}{\begin{bmatrix}x_{p} \\y_{p} \\0\end{bmatrix} = \begin{bmatrix}t_{px}^{pl} \\t_{py}^{pl} \\0\end{bmatrix}} & (16)\end{matrix}$

Accordingly, x_(w) is a function of t^(pl) _(p) and r, and F and Gbecome functions of the camera position t, camera orientation ω,position t^(pl) _(p) of the point marker on the plane, and parameter rof the plane, as in Expression (17).F(t _(x) ,t _(y) ,t _(z),ω_(x),ω_(y),ω_(z) ,t ^(pl) _(px) ,t ^(pl) _(py),r,θ,φ)=0G(t _(x) ,t _(y) ,t _(z),ω_(x),ω_(y),ω_(z) ,t ^(pl) _(px) ,t ^(pl) _(py),r,θ,φ)=0   (17)

On the other hand, in the case of square markers, in addition to theposition t^(pl) _(s)=t^(pl) _(sx) t^(pl) _(sy) O]^(t) on the planecoordinate system, the square marker constrained on the plane isrepresented with the rotation θ^(pl) around the normal line vector ofthe square marker (i.e., around the z_(p) axis of the plane coordinatesystem). So, if we say that the position of the vertices of the squaremarker in the square marker coordinate system is x_(s)=x_(s) y_(s)O]^(t), the plane coordinate x_(p) thereof is represented as shown inExpression (18). $\begin{matrix}{\begin{bmatrix}x_{p} \\y_{p} \\1\end{bmatrix} = {\begin{bmatrix}{\cos\quad\theta^{pl}} & {{- \sin}\quad\theta^{pl}} & t_{sx}^{pl} \\{\sin\quad\theta^{pl}} & {\cos\quad\theta^{pl}} & t_{sx}^{pl} \\0 & 0 & 1\end{bmatrix}\begin{bmatrix}x_{s} \\y_{s} \\1\end{bmatrix}}} & (18)\end{matrix}$

Thus, x_(w) is a function of t^(pl) _(s), θ^(pl), and r, and F and Gbecome functions of the camera position t, camera orientation ω,position t^(pl) _(s) of the square marker on the plane, orientationθ^(pl) of the square marker on the plane, and parameter r of the plane,as in Expression (19).F(t _(x) ,t _(y) ,t _(z),ω_(x),ω_(y),ω_(z) ,t ^(pl) _(sx) ,t ^(pl)_(sy),θ^(pl) ,r,θ,φ)=0G(t _(x) ,t _(y) ,t _(z),ω_(x),ω_(y),ω_(z) ,t ^(pl) _(sx) ,t ^(pl)_(sy),θ^(pl) ,r,θ,φ)=0   (19)

Expressions (10), (12), (17), and (19) are nonlinear expressionsregarding the camera position and orientation, position and orientationof the marker, and the plane parameter. Accordingly, linearization isperformed in the vicinity of the approximate values regarding theposition and orientation of the camera, the position and orientation ofthe marker, and the plane parameters using the Taylor expansion up tofirst order, and the position and orientation of the camera, theposition and orientation of the marker, and the plane parameters areobtained using repeated calculation.

Expressions (20), (21), (22), and (23) are obtained by linearizing thecorresponding Expressions (10), (12), (17), and (19) respectively. Here,Δt_(x), Δt_(y), and Δt_(z) represent the correction value for theapproximate values of the position of the camera; Δω_(x), Δω_(y), andΔω_(z) for the orientation of the camera; Δt_(px), Δt_(py), and Δt_(pz)for the position of the point marker not constrained to the plane;Δt_(sx), Δt_(sy), and Δt_(sz) for the position of the square marker notconstrained to the plane; Δω_(sx), Δω_(sy), and Δω_(sz) for theorientation of the square marker not constrained to the plane; Δt^(pl)_(px) and Δt^(pl) _(py) for the position on the plane of the pointmarker constrained to the plane; Δt^(pl) _(sx) and Δt^(pl) _(sy) for theposition on the plane of the square marker constrained to the plane;Δθ^(pl) for the orientation on the plane of the square markerconstrained with the plane; and Δr, Δθ, and Δφ for the correction valueas to the approximate values of the plane parameters. $\begin{matrix}{{{F^{0} + {\frac{\partial F}{\partial t_{x}}\Delta\quad t_{x}} + {\frac{\partial F}{\partial t_{y}}\Delta\quad t_{y}} + {\frac{\partial F}{\partial t_{z}}\Delta\quad t_{z}} + {\frac{\partial F}{\partial\omega_{x}}\Delta\quad\omega_{x}} + {\frac{\partial F}{\partial\omega_{y}}\Delta\quad\omega_{y}} + {\frac{\partial F}{\partial\omega_{z}}\Delta\quad\omega_{z}} + {\frac{\partial F}{\partial t_{px}}\Delta\quad t_{px}} + {\frac{\partial F}{\partial t_{py}}\Delta\quad t_{py}} + {\frac{\partial F}{\partial t_{pz}}\Delta\quad t_{pz}}} = 0}{{G^{0} + {\frac{\partial G}{\partial t_{x}}\Delta\quad t_{x}} + {\frac{\partial G}{\partial t_{y}}\Delta\quad t_{y}} + {\frac{\partial G}{\partial t_{z}}\Delta\quad t_{z}} + {\frac{\partial G}{\partial\omega_{x}}\Delta\quad\omega_{x}} + {\frac{\partial G}{\partial\omega_{y}}\Delta\quad\omega_{y}} + {\frac{\partial G}{\partial\omega_{z}}\Delta\quad\omega_{z}} + {\frac{\partial G}{\partial t_{px}}\Delta\quad t_{px}} + {\frac{\partial G}{\partial t_{py}}\Delta\quad t_{py}} + {\frac{\partial G}{\partial t_{pz}}\Delta\quad t_{pz}}} = 0}} & (20) \\{{{F^{0} + {\frac{\partial F}{\partial t_{x}}\Delta\quad t_{x}} + {\frac{\partial F}{\partial t_{y}}\Delta\quad t_{y}} + {\frac{\partial F}{\partial t_{z}}\Delta\quad t_{z}} + {\frac{\partial F}{\partial\omega_{x}}\Delta\quad\omega_{x}} + {\frac{\partial F}{\partial\omega_{y}}\Delta\quad\omega_{y}} + {\frac{\partial F}{\partial\omega_{z}}\Delta\quad\omega_{z}} + {\frac{\partial F}{\partial t_{sx}}\Delta\quad t_{sx}} + {\frac{\partial F}{\partial t_{sy}}\Delta\quad t_{sy}} + {\frac{\partial F}{\partial t_{sz}}\Delta\quad t_{sz}} + {\frac{\partial F}{\partial\omega_{sx}}\Delta\quad\omega_{sx}} + {\frac{\partial F}{\partial\omega_{sy}}\Delta\quad\omega_{sy}} + {\frac{\partial F}{\partial\omega_{sz}}\Delta\quad\omega_{sz}}} = 0}{{G^{0} + {\frac{\partial G}{\partial t_{x}}\Delta\quad t_{x}} + {\frac{\partial G}{\partial t_{y}}\Delta\quad t_{y}} + {\frac{\partial G}{\partial t_{z}}\Delta\quad t_{z}} + {\frac{\partial G}{\partial\omega_{x}}\Delta\quad\omega_{x}} + {\frac{\partial G}{\partial\omega_{y}}\Delta\quad\omega_{y}} + {\frac{\partial G}{\partial\omega_{z}}\Delta\quad\omega_{z}} + {\frac{\partial G}{\partial t_{sx}}\Delta\quad t_{sx}} + {\frac{\partial G}{\partial t_{sy}}\Delta\quad t_{sy}} + {\frac{\partial G}{\partial t_{sz}}\Delta\quad t_{sz}} + {\frac{\partial G}{\partial\omega_{sx}}\Delta\quad\omega_{sx}} + {\frac{\partial G}{\partial\omega_{sy}}\Delta\quad\omega_{sy}} + {\frac{\partial G}{\partial\omega_{sz}}\Delta\quad\omega_{sz}}} = 0}} & (21) \\{{{F^{0} + {\frac{\partial F}{\partial t_{x}}\Delta\quad t_{x}} + {\frac{\partial F}{\partial t_{y}}\Delta\quad t_{y}} + {\frac{\partial F}{\partial t_{z}}\Delta\quad t_{z}} + {\frac{\partial F}{\partial\omega_{x}}\Delta\quad\omega_{x}} + {\frac{\partial F}{\partial\omega_{y}}\Delta\quad\omega_{y}} + {\frac{\partial F}{\partial\quad\omega_{z}}\Delta\quad\omega_{z}} + {\frac{\partial F}{\partial t_{px}^{pl}}\Delta\quad t_{px}^{pl}} + {\frac{\partial F}{\partial t_{py}^{pl}}{\Delta t}_{py}^{pl}} + {\frac{\partial F}{\partial r}\Delta\quad r} + {\frac{\partial F}{\partial\theta}\Delta\quad\theta} + {\frac{\partial F}{\partial\phi}{\Delta\phi}}} = 0}{{G^{0} + {\frac{\partial G}{\partial t_{x}}\Delta\quad t_{x}} + {\frac{\partial G}{\partial t_{y}}\Delta\quad t_{y}} + {\frac{\partial G}{\partial t_{z}}\Delta\quad t_{z}} + {\frac{\partial G}{\partial\omega_{x}}\Delta\quad\omega_{x}} + {\frac{\partial G}{\partial\omega_{y}}\Delta\quad\omega_{y}} + {\frac{\partial G}{\partial\omega_{z}}\Delta\quad\omega_{z}} + {\frac{\partial G}{\partial t_{px}^{pl}}\Delta\quad t_{px}^{pl}} + {\frac{\partial G}{\partial t_{py}^{pl}}{\Delta t}_{py}^{pl}} + {\frac{\partial G}{\partial r}\Delta\quad r} + {\frac{\partial G}{\partial\theta}\Delta\quad\theta} + {\frac{\partial G}{\partial\phi}{\Delta\phi}}} = 0}} & (22) \\{{{F^{0} + {\frac{\partial F}{\partial t_{x}}\Delta\quad t_{x}} + {\frac{\partial F}{\partial t_{y}}\Delta\quad t_{y}} + {\frac{\partial F}{\partial t_{z}}\Delta\quad t_{z}} + {\frac{\partial F}{\partial\omega_{x}}\Delta\quad\omega_{x}} + {\frac{\partial F}{\partial\omega_{y}}\Delta\quad\omega_{y}} + {\frac{\partial F}{\partial\quad\omega_{z}}\Delta\quad\omega_{z}} + {\frac{\partial F}{\partial t_{sx}^{pl}}\Delta\quad t_{x}^{pl}} + {\frac{\partial F}{\partial t_{sy}^{pl}}{\Delta t}_{y}^{pl}} + {\frac{\partial F}{\partial\theta^{pl}}\Delta\quad\theta^{pl}} + {\frac{\partial F}{\partial r}\Delta\quad r} + {\frac{\partial F}{\partial\theta}\Delta\quad\theta} + {\frac{\partial F}{\partial\phi}{\Delta\phi}}} = 0}{{G^{0} + {\frac{\partial G}{\partial t_{x}}\Delta\quad t_{x}} + {\frac{\partial G}{\partial t_{y}}\Delta\quad t_{y}} + {\frac{\partial G}{\partial t_{z}}\Delta\quad t_{z}} + {\frac{\partial G}{\partial\omega_{x}}\Delta\quad\omega_{x}} + {\frac{\partial G}{\partial\omega_{y}}\Delta\quad\omega_{y}} + {\frac{\partial G}{\partial\omega_{z}}\Delta\quad\omega_{z}} + {\frac{\partial G}{\partial t_{sx}^{pl}}\Delta\quad t_{sx}^{pl}} + {\frac{\partial G}{\partial t_{sy}^{pl}}{\Delta t}_{sy}^{pl}} + {\frac{\partial G}{\partial\theta^{pl}}\Delta\quad\theta^{pl}} + {\frac{\partial G}{\partial r}\Delta\quad r} + {\frac{\partial G}{\partial\theta}\Delta\quad\theta} + {\frac{\partial G}{\partial\phi}{\Delta\phi}}} = 0}} & (23)\end{matrix}$

Here, F⁰ and G⁰ in Expressions (20), (21), (22), and (23) are constants,and represent the difference between the observed position u_(o) and theprojected position (calculated position) of the marker obtained from theapproximate values regarding the position and orientation of the camera,the position of the point marker or the position and orientation of thesquare marker, and the plane parameters.

Expressions (20), (21), (22), and (23) are observation equationsregarding one point marker or one vertex of a square marker observed ona certain image. In reality, multiple point markers or multiple squaremarkers are observed on multiple images, so multiple Expressions (20),(21), (22), and (23) are obtained. Accordingly, the positions andorientations of the cameras, the positions of the point markers notconstrained on the plane, the positions and orientations of the squaremarkers not constrained on the plane, the positions of the point markersconstrained on the plane, the positions and orientations of the squaremarkers constrained on the plane, and the plane parameters are obtainedby solving multiple Expressions (20), (21), (22), and (23) assimultaneous equations regarding the correction values as to theapproximate values regarding the positions and orientations of thecameras, the positions of the point markers not constrained on theplane, the positions and orientations of the square markers notconstrained on the plane, the positions of the point markers constrainedon the plane, the positions and orientations of the square markersconstrained on the plane, and the plane parameters.

FIG. 9 is a flowchart illustrating procedures for obtaining thepositions of point markers, and the positions and orientations of squaremarkers according to the present embodiment. Now, let us say thatphotographing of scenes including markers and extraction andidentification of the markers from photographed images have beencompleted. In Step S100, the approximate values regarding the positionsand orientations of the cameras taking each image, the positions of thepoint markers, the positions and orientations of the square markers, andthe plane-parameters are set. In the subsequent steps, correction valuesfor correcting the approximate values set in Step S100 are obtained. InStep S110, formulation of simultaneous equations for simultaneouslyobtaining the correction value of each approximate value is performed soas to minimize the projection error based on the approximate valuesregarding the positions and orientations of the cameras, the positionsof the point markers, the positions and orientations of the squaremarkers, and the plane parameters. In Step S120, the correction value ofeach approximate value is obtained from the simultaneous equationsformulated in Step S110. In Step S130, new approximate values areobtained by correcting each approximate value with the correspondingcorrection value obtained in Step S120. In Step S140, determination ismade as to whether or not the approximate values regarding the positionsand orientations of the cameras, the positions of the point markers, thepositions and orientations of the square markers, and the planeparameters are converged to the optimized value in Step S130. If so,processing ends. If not, processing returns to Step S110 and stepsS110-S140 are repeated. More specific descriptions regarding each stepare provided below.

In Step S100, the approximate values regarding the position andorientation of the camera taking the images, the positions of the pointmarkers, the positions and orientations of the square markers, and theplane parameters are input. Here, let us say that the number of imagestaken is N, the positions of the camera taking each image aret_(i)=[t_(ix) t_(iy) t_(iz)]^(t) (i=1, and so on through N), and theorientations thereof are ω_(i)=[ω_(ix) ω_(iy) ω_(iz)] (i=1, and so onthrough N). Also, of the point markers of which the positions need to beobtained, let us say that the number of the point markers notconstrained on the plane is K_(p1), the number of the point markersconstrained on the plane is K_(p2), and the positions of the respectivepoint markers are t_(pi)=[t_(pix) t_(piy) t_(piz)]^(t) (i=1, and so onthrough K_(p1)), t^(pl) _(pi)=[t^(pl) _(pix) t^(pl) _(piy)]t (i=1, andso on through K_(p2)). Further, of the square markers of which thepositions and orientations need to be obtained, let us say that thenumber of the square markers not constrained on the plane is K_(s1), thenumber of the square markers constrained on the plane is K_(s2), and thepositions of the respective square markers are t_(si)=[t_(six) t_(siy)t_(siz)]^(t) (i=1, and so on through K_(s1)), t^(pl) _(si)=[t^(pl)_(six) t^(pl) _(siy)]t (i=1, and so on through K_(s2)), and theorientations thereof are ω_(si)=[ω_(six) ω_(siy) ω_(siz)] (i=1, and soon through K_(s1)), θ^(pl) _(i)(i=1, and so on through K_(s2))

The approximate values of the position and orientation of each cameramay be obtained from the output value of a 6-degree-of-freedom positionand orientation sensor such as a magnetic sensor, which is mounted onthe camera, or may be obtained from the correspondence between points ofwhich the positions in the reference coordinate system are known, andthe projected position of the point on the image. In the case ofemploying a point of which the position in the reference coordinatesystem is known, both the point of which the position is known, andmarkers of which the positions to be obtained are unknown are mixed inthe scene. Alternately, the approximate values of the position andorientation of each camera may be obtained based on the later-describedapproximate values of the position and orientation of a marker.

Alternately, the approximate values of the position and orientation ofeach camera may be obtained mechanically, for example, with a motioncontrol camera, or photographing with a camera of which the position andorientation have been calibrated beforehand may be performed so as toemploy the calibration results.

The approximate values of the position and orientation of each markermay be a rough value measured by hand using a measuring tape, ruler,protractor, or surveying instrument, or may be a value estimated onceusing the method according to the present embodiment or the like. Theapproximate values of the position and orientation of each markerconstrained on the plane may be the values of the position andorientation of the marker in the plane coordinate system.

In the event that the markers on the same plane are all point markers,the approximate values of the plane parameters may be obtained byobtaining a regression plane from the approximate value of the positionof each point marker. On the other hand, in the event that the markerson the same plane include the square markers, the approximate values ofthe plane parameters may be obtained from the position and orientationof each square marker by assuming that each square marker is the plane.Further, the user may specify the plane parameters directly withoutobtaining the plane parameters from the approximate values of theposition and orientation of each marker.

Next, in Step S110, observation equations of Expressions (20), (21),(22), and (23) are formulated of a number corresponding to the number ofthe markers to be observed on the image. Expressions (20), (21), (22),and (23) are observation equations regarding the correction values ofthe position and orientation of one camera, the correction value of theposition of one point marker or the correction values of the positionand orientation of one square marker, and the correction value of oneplane parameter. Here, as shown in Expression (24), an observationequation regarding the positions and orientations of N cameras, thepositions of K_(p1) point markers not constrained on the plane, thepositions of K_(p2) point markers constrained on the plane, thepositions of K_(s1) square markers not constrained on the plane, thepositions of K_(s2) square markers constrained on the plane, and P planeparameters is formulated. In this case, the number of unknown correctionvalues is (6×N+3×K_(p1)+2×K_(p2)+6×K_(s1)+3×K_(s2)+3×P). $\begin{matrix}{{{F^{0} + {\sum\limits_{i = 1}^{N}\left( {{\frac{\partial F}{\partial t_{ix}}\Delta\quad t_{ix}} + {\frac{\partial F}{\partial t_{iy}}\Delta\quad t_{iy}} + {\frac{\partial F}{\partial t_{iz}}\Delta\quad t_{iz}} + {\frac{\partial F}{\partial\omega_{ix}}\Delta\quad\omega_{ix}} + {\frac{\partial F}{\partial\omega_{iy}}\Delta\quad\omega_{iy}} + {\frac{\partial F}{\partial\quad\omega_{iz}}\Delta\quad\omega_{iz}}} \right)} + {\sum\limits_{i = 1}^{K_{p1}}\left( {{\frac{\partial F}{\partial t_{pix}}\Delta\quad t_{pix}} + {\frac{\partial F}{\partial t_{piy}}\Delta\quad t_{piy}} + {\frac{\partial F}{\partial t_{piz}}\Delta\quad t_{piz}}} \right)} + {\sum\limits_{i = 1}^{K_{s1}}\left( {{\frac{\partial F}{\partial t_{six}}\Delta\quad t_{six}} + {\frac{\partial F}{\partial t_{siy}}\Delta\quad t_{siy}} + {\frac{\partial F}{\partial t_{siz}}\Delta\quad t_{siz}} + {\frac{\partial F}{\partial\omega_{six}}\Delta\quad\omega_{six}} + {\frac{\partial F}{\partial\omega_{siy}}\Delta\quad\omega_{siy}} + {\frac{\partial F}{\partial\quad\omega_{siz}}\Delta\quad\omega_{siz}}} \right)} + {\sum\limits_{i = 1}^{K_{p2}}\left( {{\frac{\partial F}{\partial t_{pix}^{pl}}\Delta\quad t_{pix}^{pl}} + {\frac{\partial F}{\partial t_{piy}^{pl}}\Delta\quad t_{piy}^{pl}}} \right)} + {\sum\limits_{i = 1}^{K_{s2}}\left( {{\frac{\partial F}{\partial t_{six}^{pl}}\Delta\quad t_{six}^{pl}} + {\frac{\partial F}{\partial t_{siy}^{pl}}\Delta\quad t_{siy}^{pl}} + {\frac{\partial F}{\partial\theta^{pl}}\Delta\quad\theta^{pl}}} \right)} + {\sum\limits_{i = 1}^{P}\left( {{\frac{\partial F}{\partial r}\Delta\quad r} + {\frac{\partial F}{\partial\theta}\Delta\quad\theta} + {\frac{\partial F}{\partial\phi}{\Delta\phi}}} \right)}} = 0}\begin{matrix}{{G^{0} + {\sum\limits_{i = 1}^{N}\left( {{\frac{\partial G}{\partial t_{ix}}\Delta\quad t_{ix}} + {\frac{\partial G}{\partial t_{iy}}\Delta\quad t_{iy}} + {\frac{\partial G}{\partial t_{iz}}\Delta\quad t_{iz}} + {\frac{\partial G}{\partial\omega_{ix}}\Delta\quad\omega_{ix}} + {\frac{\partial G}{\partial\omega_{iy}}\Delta\quad\omega_{iy}} + {\frac{\partial G}{\partial\quad\omega_{iz}}\Delta\quad\omega_{iz}}} \right)} + {\sum\limits_{i = 1}^{K_{p1}}\left( {{\frac{\partial G}{\partial t_{pix}}\Delta\quad t_{pix}} + {\frac{\partial G}{\partial t_{piy}}\Delta\quad t_{piy}} + {\frac{\partial G}{\partial t_{piz}}\Delta\quad t_{piz}}} \right)} + {\sum\limits_{i = 1}^{K_{s1}}\left( {{\frac{\partial G}{\partial t_{six}}\Delta\quad t_{six}} + {\frac{\partial G}{\partial t_{siy}}\Delta\quad t_{siy}} + {\frac{\partial G}{\partial t_{siz}}\Delta\quad t_{siz}} + {\frac{\partial G}{\partial\omega_{six}}\Delta\quad\omega_{six}} + {\frac{\partial G}{\partial\omega_{siy}}\Delta\quad\omega_{siy}} + {\frac{\partial G}{\partial\quad\omega_{siz}}\Delta\quad\omega_{siz}}} \right)} + {\sum\limits_{i = 1}^{K_{p2}}\left( {{\frac{\partial G}{\partial t_{pix}^{pl}}\Delta\quad t_{pix}^{pl}} + {\frac{\partial G}{\partial t_{piy}^{pl}}\Delta\quad t_{piy}^{pl}}} \right)} + {\sum\limits_{i = 1}^{K_{s2}}\left( {{\frac{\partial G}{\partial t_{six}^{pl}}\Delta\quad t_{six}^{pl}} + {\frac{\partial G}{\partial t_{siy}^{pl}}\Delta\quad t_{siy}^{pl}} + {\frac{\partial G}{\partial\theta^{pl}}\Delta\quad\theta^{pl}}} \right)} + {\sum\limits_{i = 1}^{P}\left( {{\frac{\partial G}{\partial r}\Delta\quad r} + {\frac{\partial G}{\partial\theta}\Delta\quad\theta} + {\frac{\partial G}{\partial\phi}{\Delta\phi}}} \right)}} = 0} & \quad\end{matrix}} & (24)\end{matrix}$

If we say that the number of point markers to be detected from an imagei (i=1, and so on through N) is d_(pi), and the number of square markersis d_(si), the number D_(p) of the point markers to be detected from theN images and the number D_(s) of the square markers to be detected fromthe N images are represented as shown in Expression (25).$\begin{matrix}{{D_{p} = {\sum\limits_{i = 1}^{N}d_{pi}}}{D_{s} = {\sum\limits_{i = 1}^{N}d_{si}}}} & (25)\end{matrix}$

In the case in which the number of the point markers to be detected fromthe N images is D_(p), and the number of the square markers to bedetected from the N images is D_(s), (D_(p)+4×D_(s)) sets of observationequation (24), i.e., 2×(D_(p)+4×D_(s)) observation equations areformulated. If simultaneous equations are formulated by transposing theconstant terms F⁰ and G⁰ on the left side of the Expression (24) to theright side thereof, the simultaneous equations are written in a matrixformat as shown in Expression (26).J·Δ=E   (26)

J is called a “Jacobian matrix” in which partial differentialcoefficients regarding the position and orientation of a camera, theposition of a point marker, the position and orientation of a squaremarker, and a plane parameter, of F and G are arrayed. The number ofrows of the Jacobian matrix J is the number of the observationequations, i.e., 2×(D_(p)+4×D_(s)), the number of columns thereof is thenumber of unknowns, i.e., (6×N+3×K_(p1)+2×K_(p2)+6×K_(s2)+3×K_(s2)+3×P).Δ represents a correction vector. The size of the correction vector isthe number of unknowns, i.e.,(6×N+3×K_(p1)+2×K_(p2)+6×K_(s1)+3×K_(s2)+3×P). E represents an errorvector, and has −F₀ and −G₀ which are the differences between thecalculated position of the projected position due to the approximatevalue and the observed position. The size of E is the number of theobservation equations, i.e., 2×(D_(p)+4×D_(s))

Note that the origin, scale, and orientation of the reference coordinatesystem can be clearly specified by photographing point markers of whichthe positions are known or square markers of which the positions andorientations are known in the reference coordinate systemsimultaneously. In Expression (24), the partial differentialcoefficients regarding the positions and orientations of these markersbecome zero. In order to clearly specify the origin, scale, andorientation of the reference coordinate system, three point markers, ofwhich the positions are known, need to be employed in the case of pointmarkers, or one square marker of which the position and orientation areknown needs to be employed in the case of square markers.

Next, in Step S120, the correction values as to the approximate valuesregarding the positions and orientations of the cameras, the positionsof the point markers, the positions and orientations of the squaremarkers, and the plane parameters, are obtained using Expression (26).In the case that the Jacobian matrix J is a square matrix, thecorrection value vector Δ is obtained by multiplying both sides ofExpression (26) by the inverse matrix of the matrix J. In the case thatthe matrix J is not a square matrix, the correction value vector Δ isobtained using the least mean square method, as shown in Expression(27).Δ=(J ^(t) ·J)⁻¹ J ^(t) ·E   (27)

Next, description is made regarding a method for obtaining each factorof the Jacobian matrix J. First, definition is made that F=[F G]^(t).From Expressions (7) and (8), F and G can be written as shown inExpression (28), so F and G are functions of x_(c), y_(c), and z_(c).$\begin{matrix}{{F = {{{- f}\quad\frac{x_{c}}{z_{c}}} - u_{ox}}}{G = {{{- f}\quad\frac{y_{c}}{z_{c}}} - u_{oy}}}} & (28)\end{matrix}$

The Jacobian matrix J_(FX) _(c) of which each factor has a partialdifferential coefficient according to x_(c), y_(c), and z_(c) of F andG, can be written as shown in Expression (29). $\begin{matrix}{J_{{Fx}_{c}} = {\begin{bmatrix}\frac{\partial F}{\partial x_{c}} & \frac{\partial F}{\partial y_{c}} & \frac{\partial F}{\partial z_{c}} \\\frac{\partial G}{\partial x_{c}} & \frac{\partial G}{\partial y_{c}} & \frac{\partial G}{\partial z_{c}}\end{bmatrix} = \begin{bmatrix}{- \frac{f}{z_{c}}} & 0 & \frac{{fx}_{c}}{z_{c}^{2}} \\0 & {- \frac{f}{z_{c}}} & \frac{{fy}_{c}}{z_{c}^{2}}\end{bmatrix}}} & (29)\end{matrix}$

Here, if x_(c), y_(c), z_(c) are the functions of variables S₁, S₂, andso on through S_(m), then F and G are also the functions of thevariables S₁, S₂, and so on through S_(m). The Jacobian matrix J_(Fs) ofwhich each factor has a partial differential coefficient according toS₁, S₂, and so on through S_(m) of F and G, can be decomposed as shownin Expression (30). $\begin{matrix}{J_{Fs} = {\begin{bmatrix}\frac{\partial F}{\partial s_{1}} & \frac{\partial F}{\partial s_{2}} & \cdots & \frac{\partial F}{\partial s_{m}} \\\frac{\partial G}{\partial s_{1}} & \frac{\partial G}{\partial s_{2}} & \cdots & \frac{\partial G}{\partial s_{m}}\end{bmatrix}\quad = \quad{{J_{{Fx}_{c}} \cdot J_{x_{c}s}}\quad = {\begin{bmatrix}\frac{\partial F}{\partial x_{c}} & \frac{\partial F}{\partial y_{c}} & \frac{\partial F}{\partial z_{c}} \\\frac{\partial G}{\partial x_{c}} & \frac{\partial G}{\partial y_{c}} & \frac{\partial G}{\partial z_{c}}\end{bmatrix}\begin{bmatrix}\frac{\partial x_{c}}{\partial s_{1}} & \frac{\partial x_{c}}{\partial s_{2}} & \cdots & \frac{\partial x_{c}}{\partial s_{m}} \\\frac{\partial y_{c}}{\partial s_{1}} & \frac{\partial y_{c}}{\partial s_{2}} & \cdots & \frac{\partial y_{c}}{\partial s_{m}} \\\frac{\partial z_{c}}{\partial s_{1}} & \frac{\partial z_{c}}{\partial s_{2}} & \cdots & \frac{\partial z_{c}}{\partial s_{m}}\end{bmatrix}}}}} & (30)\end{matrix}$By substituting s with the camera position, camera orientation, markerposition, marker orientation, and plane parameters, the partialdifferential coefficients regarding the camera position, cameraorientation, marker position, marker orientation, and plane parameters,of F and G can be obtained.

The Jacobian matrix J_(Ft) of which each factor has a partialdifferential coefficient according to the camera position t_(x), t_(y),and t_(z) of F and G can be written as shown in Expression (31).$\begin{matrix}{J_{Ft} = {{J_{{Fx}_{c}} \cdot J_{x_{c}t}} = {{J_{{Fx}_{c}} \cdot \begin{bmatrix}\frac{\partial x_{c}}{\partial t_{x}} & \frac{\partial x_{c}}{\partial t_{y}} & \frac{\partial x_{c}}{\partial t_{z}} \\\frac{\partial y_{c}}{\partial t_{x}} & \frac{\partial y_{c}}{\partial t_{y}} & \frac{\partial y_{c}}{\partial t_{z}} \\\frac{\partial z_{c}}{\partial t_{x}} & \frac{\partial z_{c}}{\partial t_{y}} & \frac{\partial z_{c}}{\partial t_{z}}\end{bmatrix}} = {J_{{Fx}_{c}} \cdot \begin{bmatrix}{- R_{11}} & {- R_{12}} & {- R_{13}} \\{- R_{21}} & {- R_{22}} & {- R_{23}} \\{- R_{31}} & {- R_{32}} & {- R_{33}}\end{bmatrix}}}}} & (31)\end{matrix}$

Note that the relationship between the orientation ω and 3×3 rotationtransform matrix R is as represented in Expression (5).

The Jacobian matrix J_(Fω) of which each factor has a partialdifferential coefficient according to the camera orientations ω_(x),ω_(y), and ω_(z) of F and G can be written in a decomposed manner asshown in Expression (32).J _(Fω) =J _(Fx) _(c) J _(x) _(c) _(ω) =J _(Fx) _(c) ·J _(x) _(c) _(R)·J _(Rω) =J _(Fx) _(c) ·J _(x) _(c) _(R) ·J _(Rr) _(axis) ·J _(r)_(axis) _({overscore (ω)}) J _({overscore (ω)}ω)  (32)

Here, J_(X) _(c) _(R) is the Jacobian matrix as shown in Expression(33). $\begin{matrix}{J_{x_{c}R} = {\begin{bmatrix}\frac{\partial x_{c}}{\partial R_{11}} & \frac{\partial x_{c}}{\partial R_{21}} & \frac{\partial x_{c}}{\partial R_{31}} & \frac{\partial x_{c}}{\partial R_{12}} & \frac{\partial x_{c}}{\partial R_{22}} & \frac{\partial x_{c}}{\partial R_{32}} & \frac{\partial x_{c}}{\partial R_{13}} & \frac{\partial x_{c}}{\partial R_{23}} & \frac{\partial x_{c}}{\partial R_{33}} \\\frac{\partial y_{c}}{\partial R_{11}} & \frac{\partial y_{c}}{\partial R_{21}} & \frac{\partial y_{c}}{\partial R_{31}} & \frac{\partial y_{c}}{\partial R_{12}} & \frac{\partial y_{c}}{\partial R_{22}} & \frac{\partial y_{c}}{\partial R_{32}} & \frac{\partial y_{c}}{\partial R_{13}} & \frac{\partial y_{c}}{\partial R_{23}} & \frac{\partial y_{c}}{\partial R_{33}} \\\frac{\partial z_{c}}{\partial R_{11}} & \frac{\partial z_{c}}{\partial R_{21}} & \frac{\partial z_{c}}{\partial R_{31}} & \frac{\partial z_{c}}{\partial R_{12}} & \frac{\partial z_{c}}{\partial R_{22}} & \frac{\partial z_{c}}{\partial R_{32}} & \frac{\partial z_{c}}{\partial R_{13}} & \frac{\partial z_{c}}{\partial R_{23}} & \frac{\partial z_{c}}{\partial R_{33}}\end{bmatrix}\quad = \begin{bmatrix}{x_{w} - t_{x}} & 0 & 0 & {y_{w} - t_{y}} & 0 & 0 & {z_{w} - t_{z}} & 0 & 0 \\0 & {x_{w} - t_{x}} & 0 & 0 & {y_{w} - t_{y}} & 0 & 0 & {z_{w} - t_{z}} & 0 \\0 & 0 & {x_{w} - t_{x}} & 0 & 0 & {y_{w} - t_{y}} & 0 & 0 & {z_{w} - t_{z}}\end{bmatrix}}} & (33)\end{matrix}$

Also, J_(Rr) _(axis) is the Jacobian matrix as shown in Expression (34).$\begin{matrix}\begin{matrix}{J_{{Rr}_{axis}} = \begin{bmatrix}\frac{\partial R_{11}}{\partial r_{x}} & \frac{\partial R_{11}}{\partial r_{y}} & \frac{\partial R_{11}}{\partial r_{z}} & \frac{\partial R_{11}}{\partial r_{a}} \\\frac{\partial R_{21}}{\partial r_{x}} & \frac{\partial R_{21}}{\partial r_{y}} & \frac{\partial R_{21}}{\partial r_{z}} & \frac{\partial R_{21}}{\partial r_{a}} \\\frac{\partial R_{31}}{\partial r_{x}} & \frac{\partial R_{31}}{\partial r_{y}} & \frac{\partial R_{31}}{\partial r_{z}} & \frac{\partial R_{31}}{\partial r_{a}} \\\frac{\partial R_{12}}{\partial r_{x}} & \frac{\partial R_{12}}{\partial r_{y}} & \frac{\partial R_{12}}{\partial r_{z}} & \frac{\partial R_{12}}{\partial r_{a}} \\\frac{\partial R_{22}}{\partial r_{x}} & \frac{\partial R_{22}}{\partial r_{y}} & \frac{\partial R_{22}}{\partial r_{z}} & \frac{\partial R_{22}}{\partial r_{a}} \\\frac{\partial R_{32}}{\partial r_{x}} & \frac{\partial R_{32}}{\partial r_{y}} & \frac{\partial R_{32}}{\partial r_{z}} & \frac{\partial R_{32}}{\partial r_{a}} \\\frac{\partial R_{13}}{\partial r_{x}} & \frac{\partial R_{13}}{\partial r_{y}} & \frac{\partial R_{13}}{\partial r_{z}} & \frac{\partial R_{13}}{\partial r_{a}} \\\frac{\partial R_{23}}{\partial r_{x}} & \frac{\partial R_{23}}{\partial r_{y}} & \frac{\partial R_{23}}{\partial r_{z}} & \frac{\partial R_{23}}{\partial r_{a}} \\\frac{\partial R_{33}}{\partial r_{x}} & \frac{\partial R_{33}}{\partial r_{y}} & \frac{\partial R_{33}}{\partial r_{z}} & \frac{\partial R_{33}}{\partial r_{a}}\end{bmatrix}} \\{= \begin{bmatrix}{2{r_{x}\left( {1 - {\cos\quad r_{a}}} \right)}} & 0 & 0 & {\left( {r_{x}^{2} - 1} \right)\quad\sin\quad r_{a}} \\{r_{y}\left( {1 - {\cos\quad r_{a}}} \right)} & {r_{x}\left( {1 - {\cos\quad r_{a}}} \right)} & {\sin\quad r_{a}} & {{r_{x}r_{y}\quad\sin\quad r_{a}} + {r_{z}\quad\cos\quad r_{a}}} \\{r_{z}\left( {1 - {\cos\quad r_{a}}} \right)} & {{- \sin}\quad r_{a}} & {r_{x}\left( {1 - {\cos\quad r_{a}}} \right)} & {{r_{z}r_{x}\quad\sin\quad r_{a}} - {r_{y}\quad\cos\quad r_{a}}} \\{r_{y}\left( {1 - {\cos\quad r_{a}}} \right)} & {r_{x}\left( {1 - {\cos\quad r_{a}}} \right)} & {{- \sin}\quad r_{a}} & {{r_{x}r_{y}\quad\sin\quad r_{a}} - {r_{z}\quad\cos\quad r}} \\0 & {2{r_{y}\left( {1 - {\cos\quad r_{a}}} \right)}} & 0 & {\left( {r_{y}^{2} - 1} \right)\quad\sin\quad r_{a}} \\{\sin\quad r_{a}} & {r_{z}\left( {1 - {\cos\quad r_{a}}} \right)} & {r_{y}\left( {1 - {\cos\quad r_{a}}} \right)} & {{r_{y}r_{z}\quad\sin\quad r_{a}} + {r_{x}\quad\cos\quad r_{a}}} \\{r_{z}\left( {1 - {\cos\quad r_{a}}} \right)} & {\sin\quad r_{a}} & {r_{x}\left( {1 - {\cos\quad r_{a}}} \right)} & {{r_{z}r_{x}\quad\sin\quad r_{a}} + {r_{y}\quad\cos\quad r_{a}}} \\{{- \sin}\quad r_{a}} & {r_{z}\left( {1 - {\cos\quad r_{a}}} \right)} & {r_{y}\left( {1 - {\cos\quad r_{a}}} \right)} & {{r_{y}r_{z}\quad\sin\quad r_{a}} - {r_{x}\quad\cos\quad r_{a}}} \\0 & 0 & {2{r_{z}\left( {1 - {\cos\quad r_{a}}} \right)}} & {\left( {r_{z}^{2} - 1} \right)\quad\sin\quad r_{a}}\end{bmatrix}}\end{matrix} & (34)\end{matrix}$

Also, if definition is made that {overscore (ω)}=[{overscore(ω)}_(x){overscore (ω)}_(y){overscore (ω)}_(z)θ]=[r_(a)r_(x) r_(a)r_(y)r_(a)r_(z) r_(a)], then J_(r) _(axis) _({overscore (ω)}) is the Jacobianmatrix as shown in Expression (35). $\begin{matrix}{J_{r_{axis}\varpi} = {\begin{bmatrix}\frac{\partial r_{x}}{\partial\omega_{x}} & \frac{\partial r_{x}}{\partial\omega_{y}} & \frac{\partial r_{x}}{\partial\omega_{z}} & \frac{\partial r_{x}}{\partial\theta} \\\frac{\partial r_{y}}{\partial\omega_{x}} & \frac{\partial r_{y}}{\partial\omega_{y}} & \frac{\partial r_{y}}{\partial\omega_{z}} & \frac{\partial r_{y}}{\partial\theta} \\\frac{\partial r_{z}}{\partial\omega_{x}} & \frac{\partial r_{z}}{\partial\omega_{y}} & \frac{\partial r_{z}}{\partial\omega_{z}} & \frac{\partial r_{z}}{\partial\theta} \\\frac{\partial r_{a}}{\partial\omega_{x}} & \frac{\partial r_{a}}{\partial\omega_{y}} & \frac{\partial r_{a}}{\partial\omega_{z}} & \frac{\partial r_{a}}{\partial\theta}\end{bmatrix} = \begin{bmatrix}\frac{1}{\theta} & 0 & 0 & {- \frac{\varpi_{x}}{\theta^{2}}} \\0 & \frac{1}{\theta} & 0 & {- \frac{\varpi_{y}}{\theta^{2}}} \\0 & 0 & \frac{1}{\theta} & {- \frac{\varpi_{z}}{\theta^{2}}} \\0 & 0 & 0 & 1\end{bmatrix}}} & (35)\end{matrix}$

Further, j_({overscore (ω)}ω) is the Jacobian matrix as shown inExpression (36). $\begin{matrix}{J_{\varpi\omega} = {\begin{bmatrix}\frac{\partial\omega_{x}}{\partial\omega_{x}} & \frac{\partial\omega_{x}}{\partial\omega_{y}} & \frac{\partial\omega_{x}}{\partial\omega_{z}} \\\frac{\partial\omega_{y}}{\partial\omega_{x}} & \frac{\partial\omega_{y}}{\partial\omega_{y}} & \frac{\partial\omega_{y}}{\partial\omega_{z}} \\\frac{\partial\omega_{z}}{\partial\omega_{x}} & \frac{\partial\omega_{z}}{\partial\omega_{y}} & \frac{\partial\omega_{z}}{\partial\omega_{z}} \\\frac{\partial\theta}{\partial\omega_{x}} & \frac{\partial\theta}{\partial\omega_{y}} & \frac{\partial\theta}{\partial\omega_{z}}\end{bmatrix} = \begin{bmatrix}1 & 0 & 0 \\0 & 1 & 0 \\0 & 0 & 1 \\\frac{\omega_{x}}{\theta} & \frac{\omega_{y}}{\theta} & \frac{\omega_{z}}{\theta}\end{bmatrix}}} & (36)\end{matrix}$

The Jacobian matrix J_(Ft) _(p) of which each factor has a partialdifferential coefficient according to the position t_(px), t_(py), andt_(pz) of the point marker not constrained on the plane of F and G canbe written as shown in Expression (37). $\begin{matrix}\begin{matrix}{J_{{Ft}_{p}} = {J_{{Fx}_{c}} \cdot J_{x_{c}t_{p}}}} \\{= {J_{{Fx}_{c}} \cdot \begin{bmatrix}\frac{\partial x_{c}}{\partial t_{px}} & \frac{\partial x_{c}}{\partial t_{py}} & \frac{\partial x_{c}}{\partial t_{pz}} \\\frac{\partial y_{c}}{\partial t_{px}} & \frac{\partial y_{c}}{\partial t_{py}} & \frac{\partial y_{c}}{\partial t_{pz}} \\\frac{\partial z_{c}}{\partial t_{px}} & \frac{\partial z_{c}}{\partial t_{py}} & \frac{\partial z_{c}}{\partial t_{pz}}\end{bmatrix}}} \\{= {J_{{Fx}_{c}} \cdot \begin{bmatrix}R_{11} & R_{12} & R_{13} \\R_{21} & R_{22} & R_{23} \\R_{31} & R_{32} & R_{33}\end{bmatrix}}}\end{matrix} & (37)\end{matrix}$

The Jacobian matrix J_(Ft) _(S) of which each factor has a partialdifferential coefficient according to the position t_(sx), t_(sy), andt_(sz) of the square marker not constrained on the plane of F and G canbe written as shown in Expression (38). $\begin{matrix}\begin{matrix}{J_{{Ft}_{s}} = {J_{{Fx}_{c}} \cdot J_{x_{c}x_{w}} \cdot J_{x_{w}t_{s}}}} \\{= {J_{{Fx}_{c}} \cdot \begin{bmatrix}\frac{\partial x_{c}}{\partial x_{w}} & \frac{\partial x_{c}}{\partial y_{w}} & \frac{\partial x_{c}}{\partial z_{w}} \\\frac{\partial y_{c}}{\partial x_{w}} & \frac{\partial y_{c}}{\partial y_{w}} & \frac{\partial y_{c}}{\partial z_{w}} \\\frac{\partial z_{c}}{\partial x_{w}} & \frac{\partial z_{c}}{\partial y_{w}} & \frac{\partial z_{c}}{\partial z_{w}}\end{bmatrix} \cdot \begin{bmatrix}\frac{\partial x_{w}}{\partial t_{sx}} & \frac{\partial x_{w}}{\partial t_{sy}} & \frac{\partial x_{w}}{\partial t_{sz}} \\\frac{\partial y_{w}}{\partial t_{sx}} & \frac{\partial y_{w}}{\partial t_{sy}} & \frac{\partial y_{w}}{\partial t_{sz}} \\\frac{\partial z_{w}}{\partial t_{sx}} & \frac{\partial z_{w}}{\partial t_{sy}} & \frac{\partial z_{w}}{\partial t_{sz}}\end{bmatrix}}} \\{= {J_{{Fx}_{c}} \cdot \begin{bmatrix}R_{11} & R_{12} & R_{13} \\R_{21} & R_{22} & R_{23} \\R_{31} & R_{32} & R_{33}\end{bmatrix} \cdot \begin{bmatrix}1 & 0 & 0 \\0 & 1 & 0 \\0 & 0 & 1\end{bmatrix}}}\end{matrix} & (38)\end{matrix}$

The Jacobian matrix J_(Fω) _(S) of which each factor has a partialdifferential coefficient according to the orientation ω_(s) of thesquare marker not constrained on the plane of F and G can be broken downas shown in Expression (39). $\begin{matrix}{J_{F\quad\omega_{s}} = {{J_{{Fx}_{c}} \cdot J_{x_{c}x_{w}} \cdot J_{x_{w}\omega_{s}}} = {{J_{{Fx}_{c}} \cdot J_{x_{c}x_{w}} \cdot J_{x_{w}R_{s}} \cdot J_{R_{s}\omega_{s}}} = {J_{{Fx}_{c}} \cdot J_{x_{c}x_{w}} \cdot \begin{bmatrix}x_{s} & 0 & 0 & y_{s} & 0 & 0 & 0 & 0 & 0 \\0 & x_{s} & 0 & 0 & y_{s} & 0 & 0 & 0 & 0 \\0 & 0 & x_{s} & 0 & 0 & y_{s} & 0 & 0 & 0\end{bmatrix} \cdot J_{R_{s}\omega_{s}}}}}} & (39)\end{matrix}$J_(R) _(S) _(ω) _(S) can be obtained in the same way as Expressions (34)through (36).

Next, the Jacobian matrix J_(Ft_(p)^(pl))of which each factor has a partial differential coefficient according tothe position t^(pl) _(px) and t^(pl) _(py) of the point markerconstrained on the plane of F and G can be written as shown inExpression (40). J_(x) _(C) _(x) _(W) can be obtained in the same way aswithin Expression (38). $\begin{matrix}{J_{{Ft}_{p}^{pl}} = {{J_{{Fx}_{c}} \cdot J_{x_{c}x_{w}} \cdot J_{x_{w}t_{p}^{pl}}} = {{J_{{Fx}_{c}} \cdot J_{x_{c}x_{w}} \cdot \begin{bmatrix}\frac{\partial x_{w}}{\partial t_{px}^{pl}} & \frac{\partial x_{w}}{\partial t_{py}^{pl}} \\\frac{\partial y_{w}}{\partial t_{px}^{pl}} & \frac{\partial y_{w}}{\partial t_{py}^{pl}} \\\frac{\partial z_{w}}{\partial t_{px}^{pl}} & \frac{\partial z_{w}}{\partial t_{py}^{pl}}\end{bmatrix}}\quad = {J_{{Fx}_{c}} \cdot J_{x_{c}x_{w}} \cdot \begin{bmatrix}{\cos\quad\theta} & 0 \\{\sin\quad\theta\quad\sin\quad\phi} & {\cos\quad\phi} \\{{- \sin}\quad{\theta cos}\quad\phi} & {\sin\quad\phi}\end{bmatrix}}}}} & (40)\end{matrix}$

The Jacobian matrix J_(Ft_(s)^(pl))of which each factor has a partial differential coefficient according tothe position t^(pl) _(sx) and t^(pl) _(sy) of the square markerconstrained on the plane of F and G can be written as shown inExpression (41). $\begin{matrix}{J_{{Ft}_{s}^{pl}} = {{J_{{Fx}_{c}} \cdot J_{x_{c}x_{w}} \cdot J_{x_{w}x_{p}} \cdot J_{x_{p}t_{s}^{pl}}}\quad = {{J_{{Fx}_{c}} \cdot J_{x_{c}x_{w}} \cdot \begin{bmatrix}\frac{\partial x_{w}}{\partial x_{p}} & \frac{\partial x_{w}}{\partial y_{p}} \\\frac{\partial y_{w}}{\partial x_{p}} & \frac{\partial y_{w}}{\partial y_{p}} \\\frac{\partial z_{w}}{\partial x_{p}} & \frac{\partial x_{w}}{\partial z_{p}}\end{bmatrix} \cdot \begin{bmatrix}\frac{\partial x_{p}}{\partial t_{sx}^{pl}} & \frac{\partial x_{p}}{\partial t_{sy}^{pl}} \\\frac{\partial y_{p}}{\partial t_{sx}^{pl}} & \frac{\partial y_{p}}{\partial t_{sy}^{pl}}\end{bmatrix}}\quad = {J_{{Fx}_{c}} \cdot J_{x_{c}x_{w}} \cdot \begin{bmatrix}{\cos\quad\theta} & 0 \\{\sin\quad\theta\quad\sin\quad\phi} & {\cos\quad\phi} \\{{- \sin}\quad{\theta cos}\quad\phi} & {\sin\quad\phi}\end{bmatrix} \cdot \begin{bmatrix}1 & 0 \\0 & 1\end{bmatrix}}}}} & (41)\end{matrix}$

The Jacobian matrix J_(Fθ) ^(pl) of which each factor has a partialdifferential coefficient according to the orientation θ^(pl) of thesquare marker constrained on the plane of F and G can be written asshown in Expression (42). $\begin{matrix}{J_{F\quad\theta^{pl}} = {{J_{{Fx}_{c}} \cdot J_{x_{c}x_{w}} \cdot J_{x_{w}x_{p}} \cdot J_{x_{p}\theta^{pl}}}\quad = {{J_{{Fx}_{c}} \cdot J_{x_{c}x_{w}} \cdot \begin{bmatrix}\frac{\partial x_{w}}{\partial x_{p}} & \frac{\partial x_{w}}{\partial y_{p}} \\\frac{\partial y_{w}}{\partial x_{p}} & \frac{\partial y_{w}}{\partial y_{p}} \\\frac{\partial z_{w}}{\partial x_{p}} & \frac{\partial x_{w}}{\partial z_{p}}\end{bmatrix} \cdot \begin{bmatrix}\frac{\partial x_{p}}{\partial\theta^{pl}} \\\frac{\partial y_{p}}{\partial\theta^{pl}}\end{bmatrix}}\quad = {J_{{Fx}_{c}} \cdot J_{x_{c}x_{w}} \cdot \quad\begin{bmatrix}{\cos\quad\theta} & 0 \\{\sin\quad\theta\quad\sin\quad\phi} & {\cos\quad\phi} \\{{- \sin}\quad{\theta cos}\quad\phi} & {\sin\quad\phi}\end{bmatrix} \cdot \begin{bmatrix}{{- x_{s}}\sin\quad\theta^{pl}} & {{- y_{s}}\cos\quad\theta^{pl}} \\{{- x_{s}}\cos\quad\theta^{pl}} & {{- y_{s}}\sin\quad\theta^{pl}}\end{bmatrix}}}}} & (42)\end{matrix}$

The Jacobian matrix J_(Pr) of which each factor has a partialdifferential coefficient according to the plane parameters r, θ, and φof F and G can be written as shown in Expression (43). $\begin{matrix}{J_{F\quad r} = {{J_{{Fx}_{c}} \cdot J_{x_{c}x_{w}} \cdot J_{x_{w}r}}\quad = {{J_{{Fx}_{c}} \cdot J_{x_{c}x_{w}} \cdot \begin{bmatrix}\frac{\partial x_{w}}{\partial r} & \frac{\partial x_{w}}{\partial\theta} & \frac{\partial x_{w}}{\partial\phi} \\\frac{\partial y_{w}}{\partial r} & \frac{\partial y_{w}}{\partial\theta} & \frac{\partial y_{w}}{\partial\phi} \\\frac{\partial z_{w}}{\partial r} & \frac{\partial z_{w}}{\partial\theta} & \frac{\partial z_{w}}{\partial\phi}\end{bmatrix}}\quad = {J_{{Fx}_{c}} \cdot J_{x_{c}x_{w}} \cdot \quad\left\lbrack \quad\begin{matrix}{\sin\quad\theta} & {{{{- x_{p}}\sin\quad\theta}\quad + {r\quad\cos\quad\theta}}\quad} & 0 \\{{- \cos}\quad\theta\quad\sin\quad\phi} & {{x_{p}\cos\quad\theta\quad\sin\quad\phi} + {r\quad\sin\quad\theta\quad\sin\quad\phi}} & {{x_{p}\sin\quad\theta\quad\cos\quad\phi} - {y_{p}\sin\quad\phi} - {r\quad\cos\quad\theta\quad\cos\quad\phi}} \\{\cos\quad{\theta cos}\quad\phi} & {{{- x_{p}}\cos\quad\theta\quad\cos\quad\phi} - {r\quad\sin\quad\theta\quad\cos\quad\phi}} & {{x_{p}\sin\quad\theta\quad\sin\quad\phi} + {y_{p}\cos\quad\phi} - {r\quad\cos\quad\theta\quad\sin\quad\phi}}\end{matrix} \right\rbrack}}}} & (43)\end{matrix}$

Next, in Step S140, convergence is determined. Determination ofconvergence is performed by determining a condition, e.g., whether ornot the absolute value of the correction value is less than a specifiedthreshold, whether or not the difference between the calculated positionof the projected position and the observed position is less than aspecified threshold, whether or not the difference between thecalculated position of the projected position and the observed positionbecomes the minimum value, or the like. In the event that determinationis made that convergence is satisfied, estimation of the positions andorientations of markers is completed, otherwise, the flow returns toStep S110, where simultaneous equations are formulated again based onthe corrected approximate values regarding the positions andorientations of the cameras, the positions of the point markers, thepositions and orientations of the square markers, and the planeparameters.

As described above, with the present embodiment, a great number ofimages of the scene in which markers are disposed are photographed, themarkers are detected and identified from the photographed images, themultiple markers present on the same plane are specified, correctionvalues corresponding to the approximate values regarding the positionsand orientations of the cameras taking the images, the positions andorientations of the markers not constrained on the plane, the positionsand orientations of the markers constrained on the plane, and the planeparameters, are obtained based on the marker observed positions on theimages and the constraint condition that the markers are present on thesame plane, and then the approximate values are corrected with theobtained correction values. This operating cycle for obtainingcorrection values corresponding to the approximate values and correctingthe approximate values with the obtained correction values is repeated,thus obtaining the positions and orientations of the markers so as tosatisfy the constraint condition.

Note that with the present embodiment, the plane parameters are allunknown, but the present invention is not restricted to this; i.e., theplane parameters may be known. For example, in the event that the planeparameters can be easily obtained, such as the case of the x-y plane inthe reference coordinate system, the plane parameters may be known. Inthe event that the plane parameters are known, the observation equationof the point markers constrained on the plane in Expression (22) isrepresented as shown in Expression (44). $\begin{matrix}{{{F^{0} + {\frac{\partial F}{\partial t_{x}}\Delta\quad t_{x}} + {\frac{\partial F}{\partial t_{y}}\Delta\quad t_{y}} + {\frac{\partial F}{\partial t_{z}}\Delta\quad t_{z}} + {\frac{\partial F}{\partial\omega_{x}}\Delta\quad\omega_{x}} + {\frac{\partial F}{\partial\omega_{y}}\Delta\quad\omega_{y}} + {\frac{\partial F}{\partial\omega_{z}}\Delta\quad\omega_{z}} + {\frac{\partial F}{\partial t_{px}^{pl}}\Delta\quad t_{px}^{pl}} + {\frac{\partial F}{\partial t_{py}^{pl}}\Delta\quad t_{py}^{pl}}} = 0}{{G^{0} + {\frac{\partial G}{\partial t_{x}}\Delta\quad t_{x}} + {\frac{\partial G}{\partial t_{y}}\Delta\quad t_{y}} + {\frac{\partial G}{\partial t_{z}}\Delta\quad t_{z}} + {\frac{\partial G}{\partial\omega_{x}}\Delta\quad\omega_{x}} + {\frac{\partial G}{\partial\omega_{y}}\Delta\quad\omega_{y}} + {\frac{\partial G}{\partial\omega_{z}}\Delta\quad\omega_{z}} + {\frac{\partial G}{\partial t_{px}^{pl}}\Delta\quad t_{px}^{pl}} + {\frac{\partial G}{\partial t_{py}^{pl}}\Delta\quad t_{py}^{pl}}} = 0}} & (44)\end{matrix}$

Also, the observation equation of the square markers constrained on theplane in Expression (23) is represented as shown in Expression (45).$\begin{matrix}{{{F^{0} + {\frac{\partial F}{\partial t_{x}}\Delta\quad t_{x}} + {\frac{\partial F}{\partial t_{y}}\Delta\quad t_{y}} + {\frac{\partial F}{\partial t_{z}}\Delta\quad t_{z}} + {\frac{\partial F}{\partial\omega_{x}}\Delta\quad\omega_{x}} + {\frac{\partial F}{\partial\omega_{y}}\Delta\quad\omega_{y}} + {\frac{\partial F}{\partial\omega_{z}}\Delta\quad\omega_{z}} + {\frac{\partial F}{\partial t_{sx}^{pl}}\Delta\quad t_{x}^{pl}} + {\frac{\partial F}{\partial t_{sy}^{pl}}\Delta\quad t_{y}^{pl}} + {\frac{\partial F}{\partial\theta^{pl}}\Delta\quad\theta^{pl}}} = 0}{{G^{0} + {\frac{\partial G}{\partial t_{x}}\Delta\quad t_{x}} + {\frac{\partial G}{\partial t_{y}}\Delta\quad t_{y}} + {\frac{\partial G}{\partial t_{z}}\Delta\quad t_{z}} + {\frac{\partial G}{\partial\omega_{x}}\Delta\quad\omega_{x}} + {\frac{\partial G}{\partial\omega_{y}}\Delta\quad\omega_{y}} + {\frac{\partial G}{\partial\omega_{z}}\Delta\quad\omega_{z}} + {\frac{\partial G}{\partial t_{sx}^{pl}}\Delta\quad t_{sx}^{pl}} + {\frac{\partial G}{\partial t_{sy}^{pl}}\Delta\quad t_{sy}^{pl}} + {\frac{\partial G}{\partial\theta^{pl}}\Delta\quad\theta^{pl}}} = 0}} & (45)\end{matrix}$

As described above, according to the present embodiment, in the eventthat there is a constraint condition regarding the placements ofmarkers, the positions and orientations of markers can be obtainedaccurately by obtaining the positions and orientations of the markers soas to satisfy the constraint condition.

Second Embodiment

With the first embodiment, in the event that the constraint conditionthat multiple markers are present on the same plane is given as aconstraint condition regarding the placements of markers, a method forestimating the positions and orientations of markers so as to satisfythe constraint condition has been described. With the second embodiment,description will be made regarding the case in which multiple markersare present on multiple planes, and the angle between the planes isknown.

Note that description will be made here regarding the case in which twoplanes exist, and the angle between the two planes is known, but thenumber of planes is not restricted to two.

FIG. 10 illustrates the case in which two planes constraining markersexist. As described in the first embodiment, a plane 1 is representedwith a polar coordinate expression r₁=[r₁ θ₁ φ₁] in the referencecoordinate system.

On the other hand, a plane 2 is also represented with a polar coordinateexpression r₂=[r₂ θ₂ φ₂] , and let us say that θ₂ and φ₂ represent theorientation relative to the plane 1. More specifically, let us say thatthe orientation obtained when the plane 1 is rotated around the y^(p1)axis by θ₂, and then rotated around the x_(p1) axis by φ₂ is theorientation of a plane 2. Also, r₂ is the signed distance between theorigin in the reference coordinate system and the plane 2. At this time,rotation matrixes R^(p1) and R^(p2) representing the orientations of theplane 1 and plane 2 can be written as shown in Expressions (47) and (48)respectively. $\begin{matrix}\begin{matrix}{R^{p1} = {\begin{bmatrix}R_{11}^{p1} & R_{12}^{p1} & R_{13}^{p1} \\R_{21}^{p1} & R_{22}^{p1} & R_{23}^{p1} \\R_{31}^{p1} & R_{32}^{p1} & R_{33}^{p1}\end{bmatrix}\quad = {{\begin{bmatrix}1 & 0 & 0 \\0 & {\cos\quad\phi_{1}} & {{- \sin}\quad\phi_{1}} \\0 & {\sin\quad\phi_{1}} & {\cos\quad\phi_{1}}\end{bmatrix}\begin{bmatrix}{\cos\quad\theta_{1}} & 0 & {\sin\quad\theta_{1}} \\0 & 1 & 0 \\{{- \sin}\quad\theta_{1}} & 0 & {\cos\quad\theta_{1}}\end{bmatrix}}\quad = \begin{bmatrix}{\cos\quad\theta_{1}} & 0 & {\sin\quad\theta_{1}} \\{\sin\quad\theta_{1}\sin\quad\phi_{1}} & {\cos\quad\phi_{1}} & {{- \cos}\quad\theta_{1}\sin\quad\phi_{1}} \\{{- \sin}\quad\theta_{1}\cos\quad\phi_{1}} & {\sin\quad\phi_{1}} & {\cos\quad\theta_{1}\cos\quad\phi_{1}}\end{bmatrix}}}} & \quad\end{matrix} & (47) \\{R^{p2} = {\begin{bmatrix}R_{11}^{p2} & R_{12}^{p2} & R_{13}^{p2} \\R_{21}^{p2} & R_{22}^{p2} & R_{23}^{p2} \\R_{31}^{p2} & R_{32}^{p2} & R_{33}^{p2}\end{bmatrix}\quad = {{{\begin{bmatrix}1 & 0 & 0 \\0 & {\cos\quad\phi_{2}} & {{- \sin}\quad\phi_{2}} \\0 & {\sin\quad\phi_{2}} & {\cos\quad\phi_{2}}\end{bmatrix}\begin{bmatrix}{\cos\quad\theta_{2}} & 0 & {\sin\quad\theta_{2}} \\0 & 1 & 0 \\{{- \sin}\quad\theta_{2}} & 0 & {\cos\quad\theta_{2}}\end{bmatrix}} \cdot R^{p1}}\quad = {\begin{bmatrix}{\cos\quad\theta_{2}} & 0 & {\sin\quad\theta_{2}} \\{\sin\quad\theta_{1}\quad\sin\quad\phi_{2}} & {\cos\quad\phi_{2}} & {{- \cos}\quad\theta_{1}\sin\quad\phi_{2}} \\{{- \sin}\quad\theta_{1}\cos\quad\phi_{2}} & {\sin\quad\phi_{2}} & {\cos\quad\theta_{1}\cos\quad\phi_{2}}\end{bmatrix}\quad\begin{bmatrix}{\cos\quad\theta_{1}} & 0 & {\sin\quad\theta_{1}} \\{\sin\quad\theta_{1}\quad\sin\quad\phi_{1}} & {\cos\quad\phi_{1}} & {{- \cos}\quad\theta_{1}\sin\quad\phi_{1}} \\{{- \sin}\quad\theta_{1}\cos\quad\phi_{1}} & {\sin\quad\phi_{1}} & {\cos\quad\theta_{1}\cos\quad\phi_{1}}\end{bmatrix}}}}} & (48)\end{matrix}$Here, the angle between the normal line vector of the plane 1 {rightarrow over (n₁)} and the normal line vector of the plane 2 {right arrowover (n₂)} is α, so Expression (49) holds. $\begin{matrix}{{\cos\quad\alpha} = {\frac{{\overset{->}{n}}_{1} \cdot {\overset{->}{n}}_{2}}{{{\overset{->}{n}}_{1}} \cdot {{\overset{->}{n}}_{2}}} = {{\cos\quad\theta_{2}\cos\quad\phi_{2}} = {constant}}}} & (49)\end{matrix}$

Here, φ₂ can be obtained with θ₂ and α from Expression (49).

The observation equations of point markers and square markersconstrained on the plane 1 are the same as Expressions (22) and (23)according to the first embodiment. On the other hand, the plane 2becomes the functions of r₂ and θ₂, so the observation equations ofpoint markers and square markers constrained on the plane 2 arerepresented as shown in Expression (50) and Expression (51),respectively. $\begin{matrix}{{{F^{0} + {\frac{\partial F}{\partial t_{x}}\Delta\quad t_{x}} + {\frac{\partial F}{\partial t_{y}}\Delta\quad t_{y}} + {\frac{\partial F}{\partial t_{z}}\Delta\quad t_{z}} + {\frac{\partial F}{\partial\omega_{x}}\Delta\quad\omega_{x}} + {\frac{\partial F}{\partial\omega_{y}}\Delta\quad\omega_{y}} + {\frac{\partial F}{\partial\omega_{z}}\Delta\quad\omega_{z}} + {\frac{\partial F}{\partial t_{px}^{pl}}\Delta\quad t_{px}^{pl}} + {\frac{\partial F}{\partial t_{py}^{pl}}\Delta\quad t_{py}^{pl}} + {\frac{\partial F}{\partial r_{2}}\Delta\quad r_{2}} + {\frac{\partial F}{\partial\theta_{2}}\Delta\quad\theta_{2}}} = 0}{{G^{0} + {\frac{\partial G}{\partial t_{x}}\Delta\quad t_{x}} + {\frac{\partial G}{\partial t_{y}}\Delta\quad t_{y}} + {\frac{\partial G}{\partial t_{z}}\Delta\quad t_{z}} + {\frac{\partial G}{\partial\omega_{x}}\Delta\quad\omega_{x}} + {\frac{\partial G}{\partial\omega_{y}}\Delta\quad\omega_{y}} + {\frac{\partial G}{\partial\omega_{z}}\Delta\quad\omega_{z}} + {\frac{\partial G}{\partial t_{px}^{pl}}\Delta\quad t_{px}^{pl}} + {\frac{\partial G}{\partial t_{py}^{pl}}\Delta\quad t_{py}^{pl}} + {\frac{\partial G}{\partial r_{2}}\Delta\quad r_{2}} + {\frac{\partial G}{\partial\theta_{2}}\Delta\quad\theta_{2}}} = 0}} & (50) \\{{{F^{0} + {\frac{\partial F}{\partial t_{x}}\Delta\quad t_{x}} + {\frac{\partial F}{\partial t_{y}}\Delta\quad t_{y}} + {\frac{\partial F}{\partial t_{z}}\Delta\quad t_{z}} + {\frac{\partial F}{\partial\omega_{x}}\Delta\quad\omega_{x}} + {\frac{\partial F}{\partial\omega_{y}}\Delta\quad\omega_{y}} + {\frac{\partial F}{\partial\omega_{z}}\Delta\quad\omega_{z}} + {\frac{\partial F}{\partial t_{sx}^{pl}}\Delta\quad t_{x}^{pl}} + {\frac{\partial F}{\partial t_{sy}^{pl}}\Delta\quad t_{y}^{pl}} + {\frac{\partial F}{\partial\theta^{pl}}\Delta\quad\theta^{pl}} + {\frac{\partial F}{\partial r_{2}}\Delta\quad r_{2}} + {\frac{\partial F}{\partial\theta_{2}}\Delta\quad\theta_{2}}} = 0}{{G^{0} + {\frac{\partial G}{\partial t_{x}}\Delta\quad t_{x}} + {\frac{\partial G}{\partial t_{y}}\Delta\quad t_{y}} + {\frac{\partial G}{\partial t_{z}}\Delta\quad t_{z}} + {\frac{\partial G}{\partial\omega_{x}}\Delta\quad\omega_{x}} + {\frac{\partial G}{\partial\omega_{y}}\Delta\quad\omega_{y}} + {\frac{\partial G}{\partial\omega_{z}}\Delta\quad\omega_{z}} + {\frac{\partial G}{\partial t_{sx}^{pl}}\Delta\quad t_{sx}^{pl}} + {\frac{\partial G}{\partial t_{sy}^{pl}}\Delta\quad t_{sy}^{pl}} + {\frac{\partial G}{\partial\theta^{pl}}\Delta\quad\theta^{pl}} + {\frac{\partial G}{\partial r_{2}}\Delta\quad r_{2}} + {\frac{\partial G}{\partial\theta_{2}}\Delta\quad\theta_{2}}} = 0}} & (51)\end{matrix}$

With the present embodiment, in the event of obtaining the positions andorientations of markers, simultaneous equations are formulated based onobservation equations (22), (23), (50), and (51), correction values asto the approximate values regarding the positions and orientations ofthe cameras, the positions and orientations of the markers, the threeparameters of the plane 1, and the two parameters of the plane 2 areobtained, and the approximate values are optimized using repeatedcalculation. Note that in the event of obtaining φ₂ from Expression(49), two solutions having a different sign are obtained, but of thesetwo solutions of φ₂, the one solution reducing projection error as to θ₂obtained for each repeated calculation more than the other is employedas the solution.

As described above, with the present embodiment, in the case in whichmultiple markers are present on multiple planes as a constraintcondition regarding the placements of markers, and also the anglebetween these planes is known, the positions and orientations of themarkers can be obtained so as to satisfy the constraint condition.

Third Embodiment

With the third embodiment, description will be made regarding a methodfor estimating the positions and orientations of markers in the case inwhich the normal vector directions of markers are the same, such asmultiple plane markers, as a constraint condition regarding theplacements of markers.

FIG. 11 is a diagram illustrating marker placements to which the thirdembodiment can be applied. In the drawing, let us say that four squaremarkers are disposed, and the normal vectors of the four square markers,i.e., the direction of the z_(s) axis of each square marker is the same.

The normal vector directions of the square markers will be representedwith θ and φ. Let us say that the normal vector is a vector obtained bythe unit vector of the z axis in the reference coordinate system beingrotated around the x axis by θ, and then being rotated around the y axisby φ. Also, θ and φ have the common value for the multiple squaremarkers having the same normal vector direction. If we say that therotation around the z axis of each square marker is ψ, the orientationR_(s) of the square marker as to the reference coordinate system isrepresented with Expression (52). $\begin{matrix}{R_{s} = {\begin{bmatrix}R_{s11} & R_{s12} & R_{s13} \\R_{s21} & R_{s22} & R_{s23} \\R_{s31} & R_{s32} & R_{s33}\end{bmatrix}\quad = {{\begin{bmatrix}{\cos\quad\phi} & 0 & {\sin\quad\phi} \\0 & 1 & 0 \\{{- \sin}\quad\phi} & 0 & {\cos\quad\phi}\end{bmatrix}\begin{bmatrix}1 & 0 & 0 \\0 & {\cos\quad\theta} & {{- \sin}\quad\theta} \\0 & {\sin\quad\theta} & {\cos\quad\theta}\end{bmatrix}}\quad = \begin{bmatrix}{\cos\quad\varphi} & {{- \sin}\quad\varphi} & 0 \\{\sin\quad\varphi} & {\cos\quad\varphi} & 0 \\0 & 0 & 1\end{bmatrix}}}} & (52)\end{matrix}$

If we say that the position of each square marker in the referencecoordinate system is t_(s)=[t_(sx) t_(sy) t_(sz)]^(t), the positionx_(w) in the reference coordinate system of the point x_(s)=[x_(s)y_(s)]^(t) on the square marker is represented with Expression (53).$\begin{matrix}{\begin{bmatrix}x_{w} \\y_{w} \\z_{w}\end{bmatrix} = {\begin{bmatrix}R_{s11} & R_{s12} & t_{sx} \\R_{s21} & R_{s22} & t_{sy} \\R_{s31} & R_{s32} & t_{sz}\end{bmatrix}\begin{bmatrix}x_{s} \\y_{s} \\1\end{bmatrix}}} & (53)\end{matrix}$

With the present embodiment, the position in the reference coordinatesystem of each vertex of each square marker forms the function of thenormal vector directions θ and φ, z-axial orientation of the squaremarkers ψ, and the position t_(s) in the reference coordinate system ofeach square marker. Accordingly, the observation equation of each vertexof a square marker of which the normal vector being constrained isrepresented with Expression (54). $\begin{matrix}{{{F^{0} + {\frac{\partial F}{\partial t_{x}}\Delta\quad t_{x}} + {\frac{\partial F}{\partial t_{y}}\Delta\quad t_{y}} + {\frac{\partial F}{\partial t_{z}}\Delta\quad t_{z}} + {\frac{\partial F}{\partial\omega_{x}}\Delta\quad\omega_{x}} + {\frac{\partial F}{\partial\omega_{y}}\Delta\quad\omega_{y}} + {\frac{\partial F}{\partial\omega_{z}}\Delta\quad\omega_{z}} + {\frac{\partial F}{\partial t_{sx}}\Delta\quad t_{sx}} + {\frac{\partial F}{\partial t_{sy}}\Delta\quad t_{sy}} + {\frac{\partial F}{\partial t_{sz}}\Delta\quad t_{sz}} + {\frac{\partial F}{\partial\theta}\Delta\quad\theta} + {\frac{\partial F}{\partial\phi}\Delta\quad\phi} + {\frac{\partial F}{\partial\varphi}\Delta\quad\varphi}} = 0}{{G^{0} + {\frac{\partial G}{\partial t_{x}}\Delta\quad t_{x}} + {\frac{\partial G}{\partial t_{y}}\Delta\quad t_{y}} + {\frac{\partial G}{\partial t_{z}}\Delta\quad t_{z}} + {\frac{\partial G}{\partial\omega_{x}}\Delta\quad\omega_{x}} + {\frac{\partial G}{\partial\omega_{y}}\Delta\quad\omega_{y}} + {\frac{\partial G}{\partial\omega_{z}}\Delta\quad\omega_{z}} + {\frac{\partial G}{\partial t_{sx}}\Delta\quad t_{sx}} + {\frac{\partial G}{\partial t_{sy}}\Delta\quad t_{sy}} + {\frac{\partial G}{\partial t_{sz}}\Delta\quad t_{sz}} + {\frac{\partial G}{\partial\theta}\Delta\quad\theta} + {\frac{\partial G}{\partial\phi}\Delta\quad\phi} + {\frac{\partial G}{\partial\varphi}\Delta\quad\varphi}} = 0}} & (54)\end{matrix}$

With the present embodiment, in the event of obtaining the positions andorientations of markers, simultaneous equations are formulated based onthe multiple observation equations (54), correction values as to theapproximate values regarding the positions and orientations of thecameras, the positions and orientations of markers, and the normalvector directions θ and φ are obtained, and then the approximate valuesare optimized using repeated calculation.

As described above, with the present embodiment, in the event ofmultiple markers having the common normal vector direction as aconstraint condition regarding the placements of markers, the positionsand orientations of the markers can be obtained so as to satisfy theconstraint condition.

Fourth Embodiment

With the first, second, and third embodiments, the positions andorientations of markers have been obtained under one constraintcondition regarding the placements of markers. With the fourthembodiment, a method for obtaining the positions and orientations ofmarkers under multiple constraint conditions will be described.

FIG. 12 is a diagram illustrating marker placements to which the presentembodiment can be applied. The drawing illustrates the case in which apoint marker 1 and a square marker 1 exist on the same plane (plane 1),a square marker 2 and a square marker 3 have the common normal vectordirection, and a point marker 2 and a square marker 4 have no constraintconditions regarding placement thereof.

Upon the scene illustrated in FIG. 12 being photographed with an imagingdevice, observation equation (22) holds for the point marker 1,observation equation (23) for the square marker 1, observation equation(54) for the square markers 2 and 3, observation equation (20) for thepoint marker 2, and observation equation (21) for the square marker 4.Expressions (20) through (23) and (54) are observation equationsregarding the correction value of the position and orientation of onecamera, the correction value of the position and orientation of onemarker, the correction value of one plane parameter, and the correctionvalue of one normal vector direction. With the present embodiment, asshown in Expression (55), observation equations are formulated regardingthe correction values of the positions and orientations of N cameras,the correction values of the positions of K_(p1) point markers notconstrained on the plane, the correction values of the positions ofK_(p2) point markers constrained on the plane, the correction values ofthe positions and orientations of K_(s1) square markers not constrainedon the plane, the correction values of the positions and orientations ofK_(s2) square markers constrained on the plane, the correction values ofthe positions and orientations of K_(s3) square markers of which thenormal vectors are constrained, the correction values of P planeparameters, and the correction values of V normal vector directions.${F^{0} + {\sum\limits_{i = 1}^{N}\left( {{\frac{\partial F}{\partial t_{ix}}\Delta\quad t_{ix}} + {\frac{\partial F}{\partial t_{iy}}\Delta\quad t_{iy}} + {\frac{\partial F}{\partial t_{iz}}\Delta\quad t_{iz}} + {\frac{\partial F}{\partial\omega_{ix}}{\Delta\omega}_{ix}} + {\frac{\partial F}{\partial\omega_{iy}}{\Delta\omega}_{iy}} + {\frac{\partial F}{\partial\omega_{iz}}{\Delta\omega}_{iz}}} \right)} + {\sum\limits_{i = 1}^{K_{p1}}\left( {{\frac{\partial F}{\partial t_{pix}}\Delta\quad t_{pix}} + {\frac{\partial F}{\partial t_{piy}}\Delta\quad t_{piy}} + {\frac{\partial F}{\partial t_{piz}}\Delta\quad t_{piz}}} \right)} + {\sum\limits_{i = 1}^{K_{s1}}\left( {{\frac{\partial F}{\partial t_{six}}\Delta\quad t_{six}} + {\frac{\partial F}{\partial t_{siy}}\Delta\quad t_{siy}} + {\frac{\partial F}{\partial t_{siz}}\Delta\quad t_{siz}} + {\frac{\partial F}{\partial\omega_{six}}{\Delta\omega}_{six}} + {\frac{\partial F}{\partial\omega_{siy}}{\Delta\omega}_{siy}} + {\frac{\partial F}{\partial\omega_{siz}}{\Delta\omega}_{siz}}} \right)} + {\sum\limits_{i = 1}^{K_{p2}}\left( {{\frac{\partial F}{\partial t_{pix}^{pl}}\Delta\quad t_{pix}^{pl}} + {\frac{\partial F}{\partial t_{piy}^{pl}}\Delta\quad t_{piy}^{pl}}} \right)} + {\sum\limits_{i = 1}^{K_{s2}}\left( {{\frac{\partial F}{\partial t_{six}^{pl}}\Delta\quad t_{six}^{pl}} + {\frac{\partial F}{\partial t_{siy}^{pl}}\Delta\quad t_{siy}^{pl}} + {\frac{\partial F}{\partial\theta^{pl}}{\Delta\theta}^{pl}}} \right)} + {\sum\limits_{i = 1}^{K_{s3}}\left( {{\frac{\partial F}{\partial t_{six}^{v}}\Delta\quad t_{six}^{v}} + {\frac{\partial F}{\partial t_{siy}^{v}}\Delta\quad t_{siy}^{v}} + {\frac{\partial F}{\partial t_{siy}^{v}}\Delta\quad t_{siy}^{v}} + {\frac{\partial F}{\partial\Psi}{\Delta\Psi}}} \right)} + {\sum\limits_{i = 1}^{P}\left( {{\frac{\partial F}{\partial r}\Delta\quad r} + {\frac{\partial F}{\partial\theta}{\Delta\theta}} + {\frac{\partial F}{\partial\phi}{\Delta\phi}}} \right)} + {\sum\limits_{i = 1}^{V}\left( {{\frac{\partial F}{\partial\theta^{v}}{\Delta\theta}^{v}} + {\frac{\partial F}{\partial\phi^{v}}{\Delta\phi}^{v}}} \right)}} = {{{0G^{0}} + {\sum\limits_{i = 1}^{N}\left( {{\frac{\partial G}{\partial t_{ix}}\Delta\quad t_{ix}} + {\frac{\partial G}{\partial t_{iy}}\Delta\quad t_{iy}} + {\frac{\partial G}{\partial t_{iz}}\Delta\quad t_{iz}} + {\frac{\partial G}{\partial\omega_{ix}}{\Delta\omega}_{ix}} + {\frac{\partial G}{\partial\omega_{iy}}{\Delta\omega}_{iy}} + {\frac{\partial G}{\partial\omega_{iz}}{\Delta\omega}_{iz}}} \right)} + {\sum\limits_{i = 1}^{K_{p1}}\left( {{\frac{\partial G}{\partial t_{pix}}\Delta\quad t_{pix}} + {\frac{\partial G}{\partial t_{piy}}\Delta\quad t_{piy}} + {\frac{\partial G}{\partial t_{piz}}\Delta\quad t_{piz}}} \right)} + {\sum\limits_{i = 1}^{K_{s1}}\left( {{\frac{\partial G}{\partial t_{six}}\Delta\quad t_{six}} + {\frac{\partial G}{\partial t_{siy}}\Delta\quad t_{siy}} + {\frac{\partial G}{\partial t_{siz}}\Delta\quad t_{siz}} + {\frac{\partial G}{\partial\omega_{six}}{\Delta\omega}_{six}} + {\frac{\partial G}{\partial\omega_{siy}}{\Delta\omega}_{siy}} + {\frac{\partial G}{\partial\omega_{siz}}{\Delta\omega}_{siz}}} \right)} + {\sum\limits_{i = 1}^{K_{p2}}\left( {{\frac{\partial G}{\partial t_{pix}^{pl}}\Delta\quad t_{pix}^{pl}} + {\frac{\partial G}{\partial t_{piy}^{pl}}\Delta\quad t_{piy}^{pl}}} \right)} + {\sum\limits_{i = 1}^{K_{s2}}\left( {{\frac{\partial G}{\partial t_{six}^{pl}}\Delta\quad t_{six}^{pl}} + {\frac{\partial G}{\partial t_{siy}^{pl}}\Delta\quad t_{siy}^{pl}} + {\frac{\partial G}{\partial\theta^{pl}}{\Delta\theta}^{pl}}} \right)} + {\sum\limits_{i = 1}^{K_{s3}}\left( {{\frac{\partial G}{\partial t_{six}^{v}}\Delta\quad t_{six}^{v}} + {\frac{\partial G}{\partial t_{siy}^{v}}\Delta\quad t_{siy}^{v}} + \frac{\partial G}{\partial t_{siy}^{v}} + {\frac{\partial G}{\partial\Psi}{\Delta\Psi}}} \right)} + {\sum\limits_{i = 1}^{P}\left( {{\frac{\partial G}{\partial r}\Delta\quad r} + {\frac{\partial G}{\partial\theta}{\Delta\theta}} + {\frac{\partial G}{\partial\phi}{\Delta\phi}}} \right)} + {\sum\limits_{i = 1}^{V}\left( {{\frac{\partial G}{\partial\theta^{v}}{\Delta\theta}^{v}} + {\frac{\partial G}{\partial\phi^{v}}{\Delta\phi}^{v}}} \right)}} = 0}$

Expression (55) serving as an observation equation is formulated foreach point marker, and each vertex of each square marker detected on thephotographed images, and is solved as simultaneous equations, thussimultaneously obtaining correction values corresponding to theapproximate values regarding a camera position t_(i), camera orientationω_(i), placement-unconstrained point marker position t_(pi),placement-unconstrained square marker position t_(si), orientationω_(si), on-plane-constrained point marker position t^(pl) _(pi),on-plane-constrained square marker position t^(pl) _(si), orientationθ^(pl) _(i), normal-vector-constrained square marker position t^(v)_(si), orientation ψ_(i), plane parameter r_(i), normal vectordirections θ^(v) _(i) and φ^(v) _(i), and optimizing the approximatevalues by repeating correction of the approximate values using repeatedcalculation.

A GUI (Graphical User Interface) for setting the marker placementconditions according to the present embodiment is illustrated in FIGS.15A through 15C. With the GUI in FIGS. 15A through 15C, grouping ofmarkers, and the constraint condition type of each group can be set. TheGUI in FIGS. 15A through 15C further include a function for setting aconstraint condition type in addition to the GUI in FIG. 5. Upon theuser clicking near the selecting marker (in FIG. 15A) with the rightmouse button, a dialog box (FIG. 15B) for selecting a constraintcondition is displayed. Subsequently, the user sets a constraintcondition (e.g., plane constraint, normal vector constraint, straightline constraint) as to the selected marker group, using this dialog box.Note that other formats for the user interface may be used, for example,a menu as illustrated in FIG. 15C may be employed instead of the dialogbox illustrated in FIG. 15B.

As described above, with the present embodiment, in the case in whichmultiple constraint conditions regarding the placements of markersexist, the positions and orientations of markers can be obtained so asto satisfy all of the constraint conditions. Thus, the positions andorientations of the markers can be accurately obtained.

First Modification

As illustrated in FIG. 13, a marker may be constrained on the side faceof a cylinder. With the cylinder, the origin is taken in the center ofthe cylinder coordinate system, and the cylinder coordinate system istaken as illustrated in the drawing. The orientation of the cylinder isrepresented with a rotation θ around the x axis and a rotation φ aroundthe y axis. A rotation matrix R_(cyl) representing the orientation ofthe cylinder is represented as shown in Expression (56). $\begin{matrix}{R_{cyl} = {\quad{\begin{bmatrix}R_{cyl11} & R_{cyl12} & R_{cyl13} \\R_{cyl21} & R_{cyl22} & R_{cyl23} \\R_{cyl31} & R_{cyl32} & R_{cyl33}\end{bmatrix} = {\begin{bmatrix}{\cos\quad\phi} & 0 & {\sin\quad\phi} \\0 & 1 & 0 \\{{- \sin}\quad\phi} & 0 & {\cos\quad\phi}\end{bmatrix}{\quad\begin{bmatrix}1 & 0 & 0 \\0 & {\cos\quad\theta} & {{- \sin}\quad\theta} \\0 & {\sin\quad\theta} & {\cos\quad\theta}\end{bmatrix}}}}}} & (56)\end{matrix}$

If we say that the position of the origin of the cylinder in thereference coordinate system is t^(cyl)=[t^(cyl) _(x) t^(cyl) _(y)t^(cyl) _(z)]^(t), the position x_(w) of a point x_(cyl)=[x_(cyl)y_(cyl) z_(cyl)]^(t) on the cylinder coordinate system in the referencecoordinate system is represented as shown in Expression (57).$\begin{matrix}{\begin{bmatrix}x_{w} \\y_{w} \\z_{w} \\1\end{bmatrix} = {\quad{\begin{bmatrix}R_{cyl11} & R_{cyl12} & R_{cyl13} & t_{x}^{cyl} \\R_{cyl21} & R_{cyl22} & R_{cyl23} & t_{y}^{cyl} \\R_{cyl31} & R_{cyl32} & R_{cyl33} & t_{z}^{cyl} \\0 & 0 & 0 & 1\end{bmatrix}\begin{bmatrix}x_{cyl} \\y_{cyl} \\z_{cyl} \\1\end{bmatrix}}}} & (57)\end{matrix}$

Also, a point on the side face of the cylinder is represented withExpression (58) in the cylinder coordinate system, from the rotationangle β around the z_(cyl) axis, the z_(cyl) coordinate d, and a knowncylinder radius r_(cyl). $\begin{matrix}{\begin{bmatrix}x_{cyl} \\y_{cyl} \\z_{cyl}\end{bmatrix} = \begin{bmatrix}{r_{cyl}\cos\quad\beta} \\{r_{cyl}\sin\quad\beta} \\d\end{bmatrix}} & (58)\end{matrix}$

The projected position of the marker constrained on the side face of thecylinder forms a function regarding the position and orientation of thecamera, the position and orientation of the cylinder, and the positionsβ and d of the marker on the cylinder, so the position of the markerconstrained on the side face of the cylinder can be obtained by solvingthe observation equation for obtaining these correction values.

Second Modification

As illustrated in FIG. 14, a marker may be constrained on a surface of asphere. The sphere coordinate system is taken at the center of thesphere. A sphere has no directionality, and accordingly is representedwith only the position as to the reference coordinate system. If we saythat the position of the origin of the sphere coordinate system in thereference coordinate system is t^(sph)=[t^(sph) _(x)t^(sph) _(y)t^(sph)_(z)]^(t), the position x_(w) of a pointx_(sph)=[X_(sph)y_(sph)z_(sph)]^(t) on the sphere coordinate system inthe reference coordinate system is represented as shown in Expression(59). $\begin{matrix}{\begin{bmatrix}x_{w} \\y_{w} \\z_{w}\end{bmatrix} = \begin{bmatrix}{x_{sph} + t_{x}^{sph}} \\{y_{sph} + t_{y}^{sph}} \\{z_{sph} + t_{z}^{sph}}\end{bmatrix}} & (59)\end{matrix}$

Also, a point on the surface of the sphere having a known radius r_(sph)and angles θ and φ is represented with Expression (60) in the spherecoordinate system. $\begin{matrix}{\begin{bmatrix}x_{cyl} \\y_{cyl} \\z_{cyl}\end{bmatrix} = \begin{bmatrix}{r_{sph}\cos\quad{\theta cos\phi}} \\{r_{sph}\sin\quad{\theta cos\phi}} \\{r_{sph}\sin\quad\phi}\end{bmatrix}} & (60)\end{matrix}$

The projected position of the marker constrained on the sphere surfaceforms a function regarding the position and orientation of the camera,the position of the sphere, and the positions θ and φ of the marker onthe sphere surface, so the position of the marker constrained on thesphere surface can be obtained by solving the observation equation forobtaining these correction values.

Third Modification

A marker may be constrained on a straight line. If we say that a pointwhere the straight line passes through is t_(l)=[t_(lx) t_(ly)t_(lz)]^(t), and a directional vector is d=[d_(x) d_(y) d_(z)]^(t), theposition of the marker present on the straight line in the referencecoordinate system is represented with Expression (61). Here, s is avariable representing the marker position on the straight line.$\begin{matrix}{\begin{bmatrix}x_{w} \\y_{w} \\z_{w}\end{bmatrix} = \begin{bmatrix}{t_{lx} + {sd}_{x}} \\{t_{ly} + {sd}_{y}} \\{t_{lz} + {sd}_{z}}\end{bmatrix}} & (61)\end{matrix}$

The projected position of the marker constrained on the straight lineforms a function regarding the position and orientation of the camera,the point t_(l) where the straight line passes through, the directionalvector d of the straight line, and the position s of the marker on thestraight line, so the position of the marker constrained on the straightline can be obtained by solving the observation equation for obtainingthese correction values.

Fourth Modification

With the aforementioned embodiments, a method for subjecting observationequations to Taylor expansion, and repeating correction utilizing as faras primary items has been described as a method for correcting markerplacement information. This is a method for estimating the amount ofcorrection of unknown parameters in the case of assuming thatobservation equations are subjected to linearization locally in oneprocess of repetition, and result in no error, and is a methodequivalent to a method generally known as the Newton method. The Newtonmethod is a typical method for solving a nonlinear equation usingnumerical computation, but a repeated calculation method used for theaforementioned embodiments is not restricted to this. For example, theamount of correction may be dynamically changed based on the variance ofthe amount of correction of an unknown parameter to be estimated in oneprocess, such as with the similarly-known Levenberg-Marquardt method.Alternately; a method for performing repeated calculation in light ofthe results of Taylor expansion including higher order terms may beemployed. The essence of the present invention is obtaining the mostappropriate solution using various types of markers even in the eventthat multiple types of markers having different placement informationare mixed, and accordingly, specific methodology for repeatedcalculation serving as a numerical solution never detracts from theessence of the present invention.

While the present invention has been described with reference toexemplary embodiments, it is to be understood that the invention is notlimited to the disclosed embodiments. On the contrary, the invention isintended to cover various modifications and equivalent arrangementsincluded within the spirit and scope of the appended claims. The scopeof the following claims is to be accorded the broadest interpretation soas to encompass all such modifications and equivalent structures andfunctions.

This application claims priority from Japanese Patent Application No.2004-144891 filed May 14, 2004, which is hereby incorporated byreference herein.

1. A marker placement information estimating method for estimatingspatial placement information of markers present within space under ageometric constraint, the marker placement information estimating methodcomprising: a marker detecting step for detecting markers from aphotographed image including markers; a constraint condition settingstep for setting a constraint condition of markers; a marker projectedposition calculating step for calculating a projected position where amarker is projected on an image surface based on an approximate value ofa position and orientation of an imaging unit of the photographed imageand an approximate value of placement information of the marker; acorrection value calculating step for obtaining a correction value forcorrecting the placement information of the marker so as to reduce a sumof error between a position of the marker on the image surface detectedin the marker detecting step and the projected position of the marker onthe image surface obtained in the marker projected position calculatingstep, the correction value satisfying the constraint condition ofmarkers set in the constraint condition setting step; a placementinformation correcting step for correcting the placement information ofthe marker based on the correction value obtained in the correctionvalue calculating step; and a repeated calculating step for calculatingthe placement information of the marker by repeatedly performing themarker projected position calculating step, the correction valuecalculating step, and the placement information correcting step usingthe placement information of the marker corrected in the placementinformation correcting step instead of the projected position calculatedin the marker projected position calculating step.
 2. The markerplacement information estimating method according to claim 1, whereinthe constraint condition is represented by an equation unique to theconstraint condition, and the placement information satisfying theconstraint condition is represented by a placement information parameterunique to the constraint condition.
 3. The marker placement informationestimating method according to claim 2, wherein the constraint conditionis a constraint condition that markers exist on the same unknown plane,the equation unique to the constraint condition is a plane equation, andthe placement information parameters are a position of the marker on theplane and a rotation angle of which a rotation axis is a normal vectorof the plane.
 4. The marker placement information estimating methodaccording to claim 1, wherein the markers have planar shapes.
 5. Themarker placement information estimating method according to claim 2,wherein markers present within the space are markers of which theplacement information can be defined as position and orientation,wherein the constraint condition is a constraint condition that thedirections of normal vectors of a plurality of markers are the same, theequation unique to the constraint condition is a normal vector equation,and the placement information parameters are the position of the markerand a rotation angle of which rotation axis is the normal vector.
 6. Themarker placement information estimating method according to claim 2,wherein there are a plurality of the constraint conditions, and theplacement information of the marker is estimated by obtaining theequation unique to each constraint condition and the placementinformation parameter of the marker regarding the plurality ofconstraint conditions.
 7. A marker placement information estimatingmethod for estimating spatial placement information of markers from aphotographed image, the method comprising:. a constraint conditionsetting step for setting a constraint condition that can be defined witha geometric condition relating to placement information of markers; acapturing step for capturing photographed images including markerimages; a marker detecting step for detecting markers from thephotographed images; and an estimating step for estimating the placementinformation of the markers based on positions of the markers detected onthe images and the constraint condition.
 8. The marker placementinformation estimating method according to claim 7, wherein a constraintcondition is set according to user instructions in the setting step, andwherein the user instructions include grouping of markers.
 9. The markerplacement information estimating method according to claim 8, whereinthe user instructions include selecting a type of constraint condition.10. A program for realizing the marker placement information estimatingmethod according to claim 7 using a computer.
 11. A recording medium forstoring the program according to claim
 10. 12. An information processingdevice for estimating spatial placement information of markers from aphotographed image, the device comprising: a storing unit adapted tostore a constraint condition that can be defined with a geometriccondition relating to placement information of markers; a capturing unitadapted to capture photographed images including marker images; a markerdetecting unit adapted to detect markers from the photographed images;and an estimating unit adapted to estimate the placement information ofthe markers based on positions of the markers detected on the images andthe constraint condition.